==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-58b6f5b7dd-dq2sv Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Mon, 15 Apr 2024 20:58:01 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=58b6f5b7dd skaffold.dev/run-id=0db90a2b-df67-4cea-b134-e32ba72fc114 Annotations: Status: Running IP: 10.106.46.143 IPs: IP: 10.106.46.143 Controlled By: ReplicaSet/lodemon-58b6f5b7dd Containers: lodemon: Container ID: containerd://ff4a1ea3f9a8610d68bd5582895fdb1327c0305a32138698b4c2b62001557485 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Mon, 15 Apr 2024 20:58:02 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mhdnd (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-mhdnd: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43m default-scheduler Successfully assigned xlou/lodemon-58b6f5b7dd-dq2sv to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 43m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 43m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 340.883028ms (340.88886ms including waiting) Normal Created 43m kubelet Created container lodemon Normal Started 43m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.143:8080 Press CTRL+C to quit 21:58:59 INFO 21:58:59 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:58:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:58:59 INFO [loop_until]: OK (rc = 0) 21:58:59 DEBUG --- stdout --- 21:58:59 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:58:59 DEBUG --- stderr --- 21:58:59 DEBUG 21:58:59 INFO 21:58:59 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:58:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:58:59 INFO [loop_until]: OK (rc = 0) 21:58:59 DEBUG --- stdout --- 21:58:59 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:58:59 DEBUG --- stderr --- 21:58:59 DEBUG 21:58:59 INFO 21:58:59 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:58:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:58:59 INFO [loop_until]: OK (rc = 0) 21:58:59 DEBUG --- stdout --- 21:58:59 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:58:59 DEBUG --- stderr --- 21:58:59 DEBUG 21:58:59 INFO 21:58:59 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:58:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:00 INFO [loop_until]: OK (rc = 0) 21:59:00 DEBUG --- stdout --- 21:59:00 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:00 DEBUG --- stderr --- 21:59:00 DEBUG 21:59:00 INFO 21:59:00 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:00 INFO [loop_until]: OK (rc = 0) 21:59:00 DEBUG --- stdout --- 21:59:00 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:00 DEBUG --- stderr --- 21:59:00 DEBUG 21:59:00 INFO 21:59:00 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:00 INFO [loop_until]: OK (rc = 0) 21:59:00 DEBUG --- stdout --- 21:59:00 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:00 DEBUG --- stderr --- 21:59:00 DEBUG 21:59:00 INFO 21:59:00 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:00 INFO [loop_until]: OK (rc = 0) 21:59:00 DEBUG --- stdout --- 21:59:00 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:00 DEBUG --- stderr --- 21:59:00 DEBUG 21:59:00 INFO 21:59:00 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:00 INFO [loop_until]: OK (rc = 0) 21:59:00 DEBUG --- stdout --- 21:59:00 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:00 DEBUG --- stderr --- 21:59:00 DEBUG 21:59:00 INFO 21:59:00 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:00 INFO [loop_until]: OK (rc = 0) 21:59:00 DEBUG --- stdout --- 21:59:00 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:00 DEBUG --- stderr --- 21:59:00 DEBUG 21:59:00 INFO 21:59:00 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:00 INFO [loop_until]: OK (rc = 0) 21:59:00 DEBUG --- stdout --- 21:59:00 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:00 DEBUG --- stderr --- 21:59:00 DEBUG 21:59:00 INFO 21:59:00 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:01 INFO 21:59:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:01 INFO 21:59:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:01 INFO 21:59:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:01 INFO 21:59:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:01 INFO 21:59:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:01 INFO 21:59:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:01 INFO 21:59:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:01 INFO [loop_until]: OK (rc = 0) 21:59:01 DEBUG --- stdout --- 21:59:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:01 DEBUG --- stderr --- 21:59:01 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:02 INFO [loop_until]: OK (rc = 0) 21:59:02 DEBUG --- stdout --- 21:59:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:02 DEBUG --- stderr --- 21:59:02 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:02 INFO [loop_until]: OK (rc = 0) 21:59:02 DEBUG --- stdout --- 21:59:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:02 DEBUG --- stderr --- 21:59:02 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:02 INFO [loop_until]: OK (rc = 0) 21:59:02 DEBUG --- stdout --- 21:59:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:02 DEBUG --- stderr --- 21:59:02 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:02 INFO [loop_until]: OK (rc = 0) 21:59:02 DEBUG --- stdout --- 21:59:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:02 DEBUG --- stderr --- 21:59:02 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:02 INFO [loop_until]: OK (rc = 0) 21:59:02 DEBUG --- stdout --- 21:59:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:02 DEBUG --- stderr --- 21:59:02 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:02 INFO [loop_until]: OK (rc = 0) 21:59:02 DEBUG --- stdout --- 21:59:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:02 DEBUG --- stderr --- 21:59:02 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:02 INFO [loop_until]: OK (rc = 0) 21:59:02 DEBUG --- stdout --- 21:59:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:02 DEBUG --- stderr --- 21:59:02 DEBUG 21:59:02 INFO 21:59:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:03 INFO [loop_until]: OK (rc = 0) 21:59:03 DEBUG --- stdout --- 21:59:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:03 DEBUG --- stderr --- 21:59:03 DEBUG 21:59:03 INFO 21:59:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 21:59:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:03 INFO [loop_until]: OK (rc = 0) 21:59:03 DEBUG --- stdout --- 21:59:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 21:59:03 DEBUG --- stderr --- 21:59:03 DEBUG 21:59:03 INFO Initializing monitoring instance threads 21:59:03 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 21:59:03 INFO Starting instance threads 21:59:03 INFO 21:59:03 INFO Thread started 21:59:03 INFO [loop_until]: kubectl --namespace=xlou top node 21:59:03 INFO 21:59:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:03 INFO Thread started 21:59:03 INFO [loop_until]: kubectl --namespace=xlou top pods 21:59:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG --- status code --- 21:59:03 INFO Thread started 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG --- http response --- 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG --- status code --- 21:59:03 INFO Thread started 21:59:03 DEBUG --- http response --- 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:59:03 DEBUG --- status code --- 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG ----- output ----- 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG ----- output ----- 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG { 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 DEBUG --- http response --- 21:59:03 INFO Thread started 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 DEBUG --- http response --- 21:59:03 INFO Thread started 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG --- http response --- 21:59:03 INFO [loop_until]: OK (rc = 0) 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "data": { 21:59:03 DEBUG "data": { 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG "data": { 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG { 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG { 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 INFO Thread started 21:59:03 DEBUG --- stdout --- 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:03 DEBUG "data": { 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG "data": { 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG { 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG { 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8808Mi am-7d597cf7d6-8vr8f 7m 8822Mi am-7d597cf7d6-9wz9h 6m 8672Mi ds-cts-0 12m 8355Mi ds-cts-1 13m 8449Mi ds-cts-2 11m 8607Mi ds-idrepo-0 12m 4978Mi ds-idrepo-1 15m 828Mi ds-idrepo-2 16m 784Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 54Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1673Mi 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 DEBUG ----- output ----- 21:59:03 INFO Thread started 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "result": [ 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG "result": [ 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "data": { 21:59:03 DEBUG "data": { 21:59:03 DEBUG { 21:59:03 DEBUG --- stderr --- 21:59:03 DEBUG { 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 DEBUG { 21:59:03 INFO Thread started 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG { 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "pod": "ds-cts-1" 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "data": { 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG { 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG "data": { 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG { 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713214743" 21:59:03 INFO [loop_until]: OK (rc = 0) 21:59:03 DEBUG "metric": { 21:59:03 INFO Thread started 21:59:03 DEBUG { 21:59:03 DEBUG "result": [ 21:59:03 DEBUG }, 21:59:03 DEBUG "data": { 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG { 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG "metric": { 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "data": { 21:59:03 DEBUG "data": { 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG "data": { 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG --- stdout --- 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG "pod": "ds-cts-1" 21:59:03 DEBUG "metric": { 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 DEBUG { 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG { 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG "pod": "ds-cts-1" 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG "result": [ 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "data": { 21:59:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5089Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 10616Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 52m 0% 10849Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3140Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 3589Mi 3% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 12771Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2945Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 10296Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 3001Mi 2% gke-xlou-cdm-ds-0f601751-j488 61m 0% 10131Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2570Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 83m 0% 10281Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 7144Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 5564Mi 9% 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG }, 21:59:03 DEBUG "pod": "ds-cts-1" 21:59:03 DEBUG "metric": { 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214743" 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 INFO Thread started 21:59:03 DEBUG 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG "result": [ 21:59:03 DEBUG { 21:59:03 DEBUG "data": { 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG }, 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "result": [ 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG --- stderr --- 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG "value": [ 21:59:03 DEBUG }, 21:59:03 DEBUG "instance": "10.142.0.107:9100" 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG 21:59:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214743" 21:59:03 INFO Thread started 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG { 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "data": { 21:59:03 DEBUG "value": [ 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG ----- output ----- 21:59:03 ERROR [http_cmd]: ERROR 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 21:59:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG "result": [ 21:59:03 DEBUG 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG }, 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG 21:59:03 INFO All threads has been started 127.0.0.1 - - [15/Apr/2024 21:59:03] "GET /monitoring/start HTTP/1.1" 200 - 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "instance": "10.142.0.107:9100" 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 ERROR --- status code --- 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG }, 21:59:03 DEBUG }, 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG [...] 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG "pod": "ds-idrepo-2" 21:59:03 DEBUG }, 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "result": [ 21:59:03 DEBUG 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "status": "success", 21:59:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:59:03 DEBUG "pod": "ds-idrepo-2" 21:59:03 DEBUG "data": { 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "metric": { 21:59:03 DEBUG }, 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "metric": { 21:59:03 DEBUG 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:59:03 DEBUG 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG }, 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:59:03 DEBUG 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG "data": { 21:59:03 DEBUG "data": { 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 ERROR --- http response --- 21:59:03 DEBUG }, 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "pod": "ds-idrepo-2" 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "instance": "10.142.0.107:9100" 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "data": { 21:59:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 21:59:03 DEBUG [...] 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG "value": [ 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "metric": { 21:59:03 DEBUG [...] 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "result": [ 21:59:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG "pod": "ds-idrepo-2" 21:59:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 21:59:03 DEBUG 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "result": [ 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG 21:59:03 DEBUG { 21:59:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG "data": { 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 21:59:03 DEBUG 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG "status": "success", 21:59:03 INFO [http_cmd]: http status code OK 21:59:03 DEBUG [...] 21:59:03 DEBUG }, 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG { 21:59:03 DEBUG { 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG 21:59:03 DEBUG "metric": { 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "data": { 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG { 21:59:03 DEBUG "data": { 21:59:03 DEBUG --- status code --- 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG "metric": { 21:59:03 DEBUG "metric": { 21:59:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:59:03 DEBUG 21:59:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "metric": { 21:59:03 DEBUG 21:59:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 21:59:03 DEBUG "result": [ 21:59:03 DEBUG 21:59:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG "value": [ 21:59:03 DEBUG [...] 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 21:59:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 21:59:03 DEBUG "pod": "ds-idrepo-2" 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG "pod": "ds-idrepo-2" 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG "result": [ 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG "data": { 21:59:03 DEBUG "result": [ 21:59:03 DEBUG --- http response --- 21:59:03 DEBUG 21:59:03 DEBUG "instance": "10.142.0.43:9100" 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG }, 21:59:03 DEBUG }, 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG [...] 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "metric": { 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG { 21:59:03 DEBUG "0" 21:59:03 DEBUG 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG "pod": "ds-idrepo-0" 21:59:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 21:59:03 DEBUG "instance": "10.142.0.43:9100" 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG "metric": { 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG "result": [ 21:59:03 DEBUG "metric": { 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG ----- output ----- 21:59:03 DEBUG [...] 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "0.000019356926426043348" 21:59:03 DEBUG }, 21:59:03 DEBUG }, 21:59:03 DEBUG "0" 21:59:03 DEBUG "pod": "ds-idrepo-2" 21:59:03 DEBUG } 21:59:03 DEBUG "pod": "ds-idrepo-0" 21:59:03 DEBUG { 21:59:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 21:59:03 DEBUG [...] 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG { 21:59:03 DEBUG 21:59:03 DEBUG "0" 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG ] 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "value": [ 21:59:03 DEBUG ] 21:59:03 DEBUG }, 21:59:03 DEBUG ] 21:59:03 DEBUG }, 21:59:03 DEBUG "metric": { 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG "0" 21:59:03 DEBUG "status": "success", 21:59:03 DEBUG "data": { 21:59:03 DEBUG ] 21:59:03 DEBUG "pod": "ds-idrepo-0" 21:59:03 DEBUG [...] 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG "0.028988547947144287" 21:59:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG } 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "instance": "10.142.0.107:9100" 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG "resultType": "vector", 21:59:03 DEBUG } 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG ] 21:59:03 DEBUG }, 21:59:03 DEBUG "instance": "10.142.0.43:9100" 21:59:03 DEBUG "0" 21:59:03 DEBUG "1850675200" 21:59:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 21:59:03 DEBUG 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG "4310.533333333334" 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG }, 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG "result": [ 21:59:03 DEBUG ] 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG }, 21:59:03 DEBUG "value": [ 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG }, 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG "0" 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG ] 21:59:03 DEBUG "pod": "ds-idrepo-0" 21:59:03 DEBUG { 21:59:03 DEBUG } 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG ] 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG [...] 21:59:03 DEBUG } 21:59:03 DEBUG }, 21:59:03 DEBUG "metric": { 21:59:03 DEBUG } 21:59:03 DEBUG "0" 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG "11556.433333333332" 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "instance": "10.142.0.107:9100" 21:59:03 DEBUG }, 21:59:03 DEBUG "pod": "ds-idrepo-0" 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 21:59:03 DEBUG "0" 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG "pod": "ds-idrepo-0" 21:59:03 DEBUG } 21:59:03 DEBUG "0" 21:59:03 DEBUG ] 21:59:03 DEBUG [...] 21:59:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 21:59:03 DEBUG ] 21:59:03 DEBUG }, 21:59:03 DEBUG [...] 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG ] 21:59:03 DEBUG "value": [ 21:59:03 DEBUG }, 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG }, 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG }, 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG }, 21:59:03 DEBUG } 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "0" 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG "0" 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG 21:59:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG "0" 21:59:03 DEBUG ] 21:59:03 DEBUG "instance": "10.142.0.43:9100" 21:59:03 DEBUG }, 21:59:03 DEBUG } 21:59:03 DEBUG 21:59:03 DEBUG "0" 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG } 21:59:03 DEBUG "pod": "ds-cts-2" 21:59:03 DEBUG "1" 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG }, 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG [...] 21:59:03 DEBUG ] 21:59:03 DEBUG "0" 21:59:03 DEBUG } 21:59:03 DEBUG "0" 21:59:03 DEBUG } 21:59:03 DEBUG }, 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG "value": [ 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG "0.078" 21:59:03 DEBUG 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG "0" 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG "13848835072" 21:59:03 DEBUG ] 21:59:03 DEBUG 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG "instance": "10.142.0.43:9100" 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG }, 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG "value": [ 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG 1713214743, 21:59:03 DEBUG } 21:59:03 DEBUG } 21:59:03 DEBUG "0.000500000000000019" 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG ] 21:59:03 DEBUG } 21:59:03 DEBUG } 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [loop_until]: kubectl --namespace=xlou top pods 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713214803" 22:00:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214803" 22:00:03 INFO 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [loop_until]: kubectl --namespace=xlou top node 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG --- status code --- 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 ERROR [http_cmd]: ERROR 22:00:03 DEBUG --- status code --- 22:00:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG --- status code --- 22:00:03 ERROR --- status code --- 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG --- http response --- 22:00:03 ERROR --- http response --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG ----- output ----- 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG { 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG --- http response --- 22:00:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "status": "success", 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713214803" 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG { 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "data": { 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214803" 22:00:03 DEBUG "data": { 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 DEBUG "status": "success", 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "data": { 22:00:03 DEBUG { 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG "data": { 22:00:03 DEBUG "data": { 22:00:03 DEBUG "data": { 22:00:03 DEBUG "data": { 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "resultType": "vector", 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 DEBUG "data": { 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 DEBUG "data": { 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "data": { 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713214803" 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "resultType": "vector", 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "result": [ 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG "data": { 22:00:03 DEBUG "resultType": "vector", 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "result": [ 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG "result": [ 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG "result": [ 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG { 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "result": [ 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG "data": { 22:00:03 DEBUG "data": { 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG { 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG { 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG "metric": { 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG "result": [ 22:00:03 DEBUG { 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:00:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:00:03 INFO [loop_until]: OK (rc = 0) 22:00:03 DEBUG "metric": { 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG "metric": { 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG "pod": "ds-idrepo-2" 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:00:03 DEBUG "pod": "ds-idrepo-2" 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:00:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG --- stdout --- 22:00:03 DEBUG "instance": "10.142.0.107:9100" 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 INFO [http_cmd]: http status code OK 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:00:03 DEBUG "instance": "10.142.0.107:9100" 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG }, 22:00:03 DEBUG "metric": { 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG { 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG { 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2286m 8819Mi am-7d597cf7d6-8vr8f 2561m 8824Mi am-7d597cf7d6-9wz9h 1527m 8677Mi ds-cts-0 963m 8361Mi ds-cts-1 823m 8457Mi ds-cts-2 1032m 8615Mi ds-idrepo-0 4504m 4989Mi ds-idrepo-1 1034m 900Mi ds-idrepo-2 860m 1040Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 1026Mi idm-b85f9d978-rl76f 5m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 61Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 3623m 2085Mi 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG --- status code --- 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG }, 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:00:03 DEBUG "value": [ 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG }, 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG "metric": { 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG --- stderr --- 22:00:03 DEBUG { 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:00:03 DEBUG { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG { 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG }, 22:00:03 DEBUG { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:00:03 DEBUG { 22:00:03 DEBUG "instance": "10.142.0.107:9100" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG --- http response --- 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG }, 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG }, 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "data": { 22:00:03 DEBUG "data": { 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:00:03 DEBUG 22:00:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:00:03 DEBUG "data": { 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG 22:00:03 DEBUG "data": { 22:00:03 DEBUG 22:00:03 DEBUG "data": { 22:00:03 DEBUG "data": { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "data": { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG 22:00:03 DEBUG [...] 22:00:03 DEBUG [...] 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG 22:00:03 DEBUG ----- output ----- 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "data": { 22:00:03 DEBUG [...] 22:00:03 DEBUG "data": { 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG 22:00:03 DEBUG [...] 22:00:03 DEBUG [...] 22:00:03 DEBUG [...] 22:00:03 DEBUG "data": { 22:00:03 DEBUG [...] 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG [...] 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "result": [ 22:00:03 DEBUG { 22:00:03 DEBUG [...] 22:00:03 DEBUG { 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG 22:00:03 DEBUG "result": [ 22:00:03 DEBUG [...] 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "result": [ 22:00:03 DEBUG 22:00:03 DEBUG "result": [ 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG "data": { 22:00:03 DEBUG "status": "success", 22:00:03 DEBUG { 22:00:03 DEBUG "result": [ 22:00:03 DEBUG 22:00:03 DEBUG "result": [ 22:00:03 DEBUG [...] 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG { 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "result": [ 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "data": { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG "metric": { 22:00:03 DEBUG 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "metric": { 22:00:03 DEBUG [...] 22:00:03 DEBUG "metric": { 22:00:03 DEBUG [...] 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG { 22:00:03 DEBUG 22:00:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:00:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:00:03 DEBUG 22:00:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:00:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "resultType": "vector", 22:00:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "pod": "ds-idrepo-0" 22:00:03 DEBUG "metric": { 22:00:03 DEBUG 22:00:03 DEBUG "pod": "ds-idrepo-2" 22:00:03 DEBUG 22:00:03 DEBUG "instance": "10.142.0.107:9100" 22:00:03 DEBUG "pod": "ds-idrepo-2" 22:00:03 DEBUG 22:00:03 DEBUG "pod": "ds-idrepo-2" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "pod": "ds-idrepo-0" 22:00:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG "instance": "10.142.0.43:9100" 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG "result": [ 22:00:03 DEBUG "instance": "10.142.0.43:9100" 22:00:03 DEBUG "result": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "pod": "ds-idrepo-2" 22:00:03 DEBUG }, 22:00:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:00:03 DEBUG 22:00:03 DEBUG }, 22:00:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG 22:00:03 DEBUG }, 22:00:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:00:03 DEBUG 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG "instance": "10.142.0.107:9100" 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG { 22:00:03 DEBUG }, 22:00:03 DEBUG { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "instance": "10.142.0.43:9100" 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "metric": { 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "metric": { 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "0" 22:00:03 DEBUG "0" 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "pod": "ds-idrepo-2" 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "0" 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG }, 22:00:03 DEBUG 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "0" 22:00:03 DEBUG "0" 22:00:03 DEBUG 22:00:03 DEBUG "2229006336" 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG "0.000333333333333341" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG }, 22:00:03 DEBUG "0.0005333333333330377" 22:00:03 DEBUG }, 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG ] 22:00:03 DEBUG 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 22:00:03 DEBUG "8392.5" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG "560" 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG "value": [ 22:00:03 DEBUG ] 22:00:03 DEBUG "value": [ 22:00:03 DEBUG [...] 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG "0.0010592161224890462" 22:00:03 DEBUG [...] 22:00:03 DEBUG ] 22:00:03 DEBUG [...] 22:00:03 DEBUG [...] 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG [...] 22:00:03 DEBUG ] 22:00:03 DEBUG "11089.199999999997" 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG [...] 22:00:03 DEBUG [...] 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG [...] 22:00:03 DEBUG ] 22:00:03 DEBUG [...] 22:00:03 DEBUG ] 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG "560.7333333333333" 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG [...] 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG ] 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG ] 22:00:03 DEBUG 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG [...] 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG "pod": "ds-idrepo-0" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG [...] 22:00:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:00:03 DEBUG 22:00:03 DEBUG 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG }, 22:00:03 DEBUG "instance": "10.142.0.43:9100" 22:00:03 DEBUG "pod": "ds-idrepo-0" 22:00:03 DEBUG ] 22:00:03 DEBUG "pod": "ds-idrepo-0" 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 22:00:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:00:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:00:03 DEBUG }, 22:00:03 DEBUG 22:00:03 DEBUG }, 22:00:03 DEBUG "pod": "ds-idrepo-0" 22:00:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG }, 22:00:03 DEBUG } 22:00:03 DEBUG }, 22:00:03 DEBUG } 22:00:03 DEBUG "instance": "10.142.0.43:9100" 22:00:03 DEBUG }, 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "value": [ 22:00:03 DEBUG } 22:00:03 DEBUG "value": [ 22:00:03 DEBUG }, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "10318.392985670353" 22:00:03 DEBUG 22:00:03 DEBUG }, 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "0" 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "value": [ 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG ] 22:00:03 DEBUG "pod": "ds-cts-2" 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "0" 22:00:03 DEBUG ] 22:00:03 DEBUG "2659.0333333333333" 22:00:03 DEBUG "0" 22:00:03 DEBUG "0" 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG "2121166.786625663" 22:00:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:00:03 DEBUG "1.6497328298973246" 22:00:03 DEBUG } 22:00:03 DEBUG }, 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG "13851563008" 22:00:03 DEBUG ] 22:00:03 DEBUG }, 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG "value": [ 22:00:03 DEBUG "560.5333333333333" 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG "value": [ 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG 1713214803, 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG "0" 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG "0.078" 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG ] 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 DEBUG } 22:00:03 INFO [loop_until]: OK (rc = 0) 22:00:03 DEBUG --- stdout --- 22:00:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5086Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1209m 4% 10607Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2356m 7% 10883Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 6044m 20% 3051Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1011m 3% 3739Mi 3% gke-xlou-cdm-default-pool-1b0769c8-wpwl 1534m 5% 12748Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2943Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 4371m 14% 10311Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1269m 4% 3218Mi 2% gke-xlou-cdm-ds-0f601751-j488 1360m 4% 10132Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2439Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 4129m 13% 10296Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 3977m 13% 7187Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 3420m 21% 5965Mi 10% 22:00:03 DEBUG --- stderr --- 22:00:03 DEBUG 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 ERROR [http_cmd]: ERROR 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [loop_until]: kubectl --namespace=xlou top pods 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 ERROR --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG --- status code --- 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG --- status code --- 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 ERROR --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG { 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG { 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "data": { 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG "data": { 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG "data": { 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "data": { 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "result": [ 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG "result": [ 22:01:03 DEBUG { 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG { 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "result": [ 22:01:03 DEBUG { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "metric": { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG "result": [ 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG "result": [ 22:01:03 DEBUG { 22:01:03 DEBUG "result": [ 22:01:03 DEBUG { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG { 22:01:03 DEBUG "instance": "10.142.0.107:9100" 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "pod": "ds-idrepo-2" 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "instance": "10.142.0.107:9100" 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "metric": { 22:01:03 DEBUG }, 22:01:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:01:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:01:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:01:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "pod": "ds-idrepo-2" 22:01:03 DEBUG "pod": "ds-idrepo-2" 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "pod": "ds-idrepo-2" 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:01:03 DEBUG }, 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "result": [ 22:01:03 DEBUG }, 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG "instance": "10.142.0.107:9100" 22:01:03 DEBUG }, 22:01:03 DEBUG "instance": "10.142.0.107:9100" 22:01:03 DEBUG }, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG "value": [ 22:01:03 DEBUG { 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:01:03 DEBUG "pod": "ds-idrepo-2" 22:01:03 DEBUG 22:01:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713214863" 22:01:03 DEBUG 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214863" 22:01:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214863" 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG [...] 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "instance": "10.142.0.43:9100" 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "pod": "ds-idrepo-0" 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG "pod": "ds-idrepo-0" 22:01:03 DEBUG [...] 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG "instance": "10.142.0.43:9100" 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG "pod": "ds-idrepo-0" 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG "pod": "ds-cts-2" 22:01:03 DEBUG 22:01:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:01:03 DEBUG }, 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:01:03 INFO [http_cmd]: http status code OK 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG "instance": "10.142.0.43:9100" 22:01:03 DEBUG }, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG "0.0003333333333330302" 22:01:03 DEBUG }, 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG }, 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG --- status code --- 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "instance": "10.142.0.43:9100" 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG { 22:01:03 DEBUG 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "pod": "ds-idrepo-0" 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG ] 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "0" 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG { 22:01:03 DEBUG "value": [ 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:01:03 DEBUG { 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG }, 22:01:03 DEBUG "0" 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "0" 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG "0" 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG "value": [ 22:01:03 DEBUG } 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG ] 22:01:03 DEBUG "0" 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG --- http response --- 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG "0" 22:01:03 DEBUG "value": [ 22:01:03 DEBUG ] 22:01:03 DEBUG "0" 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "2328936448" 22:01:03 DEBUG ] 22:01:03 DEBUG "data": { 22:01:03 DEBUG }, 22:01:03 DEBUG ] 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "2299.9666666666667" 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG }, 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG ] 22:01:03 DEBUG "0" 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG "data": { 22:01:03 DEBUG "0.078" 22:01:03 DEBUG "data": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:01:03 DEBUG "data": { 22:01:03 DEBUG ] 22:01:03 DEBUG 1713214863, 22:01:03 INFO [loop_until]: OK (rc = 0) 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "value": [ 22:01:03 DEBUG } 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG ] 22:01:03 DEBUG { 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "0" 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG ] 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG ----- output ----- 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG } 22:01:03 DEBUG "3045.866666666667" 22:01:03 DEBUG --- stdout --- 22:01:03 DEBUG ] 22:01:03 DEBUG "11045.099999999999" 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG "result": [ 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG ] 22:01:03 DEBUG "0" 22:01:03 DEBUG ] 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG "result": [ 22:01:03 DEBUG } 22:01:03 DEBUG "result": [ 22:01:03 DEBUG "result": [ 22:01:03 DEBUG { 22:01:03 DEBUG "result": [ 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9817m 8556Mi am-7d597cf7d6-8vr8f 9184m 8330Mi am-7d597cf7d6-9wz9h 12346m 8696Mi ds-cts-0 5993m 8383Mi ds-cts-1 4621m 8498Mi ds-cts-2 4526m 8653Mi ds-idrepo-0 12009m 4946Mi ds-idrepo-1 1954m 1451Mi ds-idrepo-2 1993m 1440Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 7m 1026Mi idm-b85f9d978-rl76f 6m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 61Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6738m 2131Mi 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG { 22:01:03 DEBUG "116369.5" 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG "data": { 22:01:03 DEBUG "2299.9" 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG { 22:01:03 DEBUG ] 22:01:03 DEBUG { 22:01:03 DEBUG { 22:01:03 DEBUG "status": "success", 22:01:03 DEBUG { 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG --- stderr --- 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG "metric": { 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG "metric": { 22:01:03 DEBUG } 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG "data": { 22:01:03 DEBUG "metric": { 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG 22:01:03 DEBUG } 22:01:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG "result": [ 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG "pod": "ds-idrepo-2" 22:01:03 DEBUG } 22:01:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:01:03 DEBUG "pod": "ds-idrepo-2" 22:01:03 DEBUG "resultType": "vector", 22:01:03 DEBUG "result": [ 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG }, 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG { 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG "instance": "10.142.0.107:9100" 22:01:03 DEBUG { 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG "value": [ 22:01:03 DEBUG } 22:01:03 DEBUG "metric": { 22:01:03 DEBUG } 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG "metric": { 22:01:03 DEBUG 22:01:03 DEBUG } 22:01:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:01:03 DEBUG } 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG [...] 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG [...] 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG 22:01:03 DEBUG "pod": "ds-idrepo-0" 22:01:03 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:01:03 DEBUG "pod": "ds-idrepo-0" 22:01:03 DEBUG 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG }, 22:01:03 DEBUG "instance": "10.142.0.43:9100" 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "value": [ 22:01:03 DEBUG }, 22:01:03 DEBUG 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG "value": [ 22:01:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:01:03 DEBUG "2299.2" 22:01:03 DEBUG }, 22:01:03 DEBUG "0" 22:01:03 DEBUG "517.3513530581957" 22:01:03 DEBUG "0" 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG }, 22:01:03 DEBUG ] 22:01:03 DEBUG "value": [ 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG "13853930496" 22:01:03 DEBUG "value": [ 22:01:03 DEBUG } 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG 1713214863, 22:01:03 DEBUG ] 22:01:03 DEBUG "6.695783026391707" 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG "0" 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG ] 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 DEBUG } 22:01:03 INFO 22:01:03 INFO [loop_until]: kubectl --namespace=xlou top node 22:01:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:01:03 INFO [loop_until]: OK (rc = 0) 22:01:03 DEBUG --- stdout --- 22:01:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5087Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4639m 15% 10701Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10191m 34% 10609Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17489m 58% 3043Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2091m 7% 4250Mi 3% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6778m 22% 12738Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2951Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 9969m 33% 9830Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 2061m 6% 3657Mi 3% gke-xlou-cdm-ds-0f601751-j488 4572m 15% 10235Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 13253m 44% 10318Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12131m 40% 7125Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6895m 43% 6008Mi 10% 22:01:03 DEBUG --- stderr --- 22:01:03 DEBUG 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713214923" 22:02:03 ERROR [http_cmd]: ERROR 22:02:03 ERROR --- status code --- 22:02:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:02:03 ERROR --- http response --- 22:02:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:02:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG --- status code --- 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214923" 22:02:03 DEBUG --- status code --- 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG --- status code --- 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG --- http response --- 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG { 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG { 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG { 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "status": "success", 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG { 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG "data": { 22:02:03 DEBUG { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "data": { 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "data": { 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "data": { 22:02:03 DEBUG "data": { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "data": { 22:02:03 DEBUG "data": { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG "data": { 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG "data": { 22:02:03 DEBUG "data": { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "data": { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "data": { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "result": [ 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG "result": [ 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG "data": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG { 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG { 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:02:03 DEBUG { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:02:03 DEBUG "metric": { 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "pod": "ds-idrepo-2" 22:02:03 DEBUG "result": [ 22:02:03 DEBUG }, 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "pod": "ds-idrepo-2" 22:02:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "pod": "ds-idrepo-2" 22:02:03 DEBUG "pod": "ds-idrepo-2" 22:02:03 DEBUG }, 22:02:03 DEBUG "instance": "10.142.0.107:9100" 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "data": { 22:02:03 DEBUG "data": { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "pod": "ds-idrepo-2" 22:02:03 DEBUG }, 22:02:03 DEBUG { 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:02:03 DEBUG }, 22:02:03 DEBUG "pod": "ds-idrepo-2" 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG }, 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "data": { 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "metric": { 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG [...] 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "instance": "10.142.0.107:9100" 22:02:03 DEBUG "instance": "10.142.0.107:9100" 22:02:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG "pod": "ds-cts-2" 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG "pod": "ds-idrepo-0" 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:02:03 DEBUG }, 22:02:03 DEBUG "pod": "ds-idrepo-0" 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "pod": "ds-idrepo-0" 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG 22:02:03 DEBUG "pod": "ds-cts-0" 22:02:03 INFO 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "instance": "10.142.0.43:9100" 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "instance": "10.142.0.43:9100" 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "2331643904" 22:02:03 DEBUG }, 22:02:03 INFO [loop_until]: kubectl --namespace=xlou top pods 22:02:03 DEBUG "0" 22:02:03 DEBUG "value": [ 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG }, 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG "0" 22:02:03 DEBUG 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG "0" 22:02:03 DEBUG "value": [ 22:02:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:02:03 DEBUG ] 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG "0" 22:02:03 DEBUG "value": [ 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 DEBUG "0" 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "pod": "ds-idrepo-0" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214923" 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG } 22:02:03 DEBUG "0" 22:02:03 DEBUG ] 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG ] 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214923" 22:02:03 DEBUG "value": [ 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214923" 22:02:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713214923" 22:02:03 DEBUG 22:02:03 DEBUG "instance": "10.142.0.43:9100" 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG "pod": "ds-idrepo-0" 22:02:03 DEBUG "0.0003333333333333262" 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG "25841043.189017586" 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG "11256.93333333333" 22:02:03 DEBUG } 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG 1713214923, 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG }, 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG "0" 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG }, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG "value": [ 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG ] 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG }, 22:02:03 DEBUG "0" 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG } 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG "0.00033333333333350387" 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG "0" 22:02:03 DEBUG ] 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG ] 22:02:03 DEBUG "254019.5" 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:02:03 DEBUG "value": [ 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 INFO [http_cmd]: http status code OK 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG --- status code --- 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG { 22:02:03 DEBUG "2304.4" 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG ] 22:02:03 DEBUG { 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:02:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG ] 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG } 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG --- http response --- 22:02:03 DEBUG "data": { 22:02:03 DEBUG "data": { 22:02:03 DEBUG } 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG "data": { 22:02:03 DEBUG } 22:02:03 DEBUG "data": { 22:02:03 DEBUG "data": { 22:02:03 DEBUG } 22:02:03 DEBUG "data": { 22:02:03 DEBUG { 22:02:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG } 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG ] 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG ----- output ----- 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "data": { 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG } 22:02:03 DEBUG "metric": { 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG "status": "success", 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "data": { 22:02:03 DEBUG } 22:02:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "data": { 22:02:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:02:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG }, 22:02:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:02:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:02:03 DEBUG "instance": "10.142.0.107:9100" 22:02:03 DEBUG { 22:02:03 DEBUG "resultType": "vector", 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG "metric": { 22:02:03 DEBUG "result": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG { 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "pod": "ds-idrepo-2" 22:02:03 DEBUG { 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "metric": { 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG "metric": { 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "instance": "10.142.0.107:9100" 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG [...] 22:02:03 DEBUG [...] 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG [...] 22:02:03 DEBUG [...] 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:02:03 DEBUG 22:02:03 DEBUG 22:02:03 DEBUG [...] 22:02:03 DEBUG "instance": "10.142.0.43:9100" 22:02:03 DEBUG 22:02:03 DEBUG "pod": "ds-cts-0" 22:02:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:02:03 DEBUG [...] 22:02:03 DEBUG }, 22:02:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:02:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:02:03 DEBUG 22:02:03 DEBUG }, 22:02:03 DEBUG [...] 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG }, 22:02:03 DEBUG }, 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG "value": [ 22:02:03 DEBUG "value": [ 22:02:03 DEBUG 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG 22:02:03 DEBUG "0" 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG "pod": "ds-idrepo-0" 22:02:03 DEBUG "3143.6" 22:02:03 DEBUG 22:02:03 DEBUG "707.821879021879" 22:02:03 DEBUG "2304.4" 22:02:03 DEBUG "instance": "10.142.0.43:9100" 22:02:03 DEBUG ] 22:02:03 DEBUG "0.078" 22:02:03 DEBUG "2304.6666666666665" 22:02:03 DEBUG }, 22:02:03 DEBUG ] 22:02:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG }, 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG "value": [ 22:02:03 DEBUG } 22:02:03 DEBUG }, 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG "value": [ 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG ] 22:02:03 DEBUG "value": [ 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG ] 22:02:03 DEBUG "0" 22:02:03 DEBUG } 22:02:03 DEBUG 1713214923, 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG "13854585856" 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG "6.7002328509806475" 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG ] 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 DEBUG } 22:02:03 INFO [loop_until]: OK (rc = 0) 22:02:03 DEBUG --- stdout --- 22:02:03 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9966m 8557Mi am-7d597cf7d6-8vr8f 9397m 8330Mi am-7d597cf7d6-9wz9h 12543m 8695Mi ds-cts-0 6306m 8426Mi ds-cts-1 4567m 8491Mi ds-cts-2 4477m 8659Mi ds-idrepo-0 11887m 4946Mi ds-idrepo-1 1872m 1641Mi ds-idrepo-2 1904m 1619Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 1m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6713m 2134Mi 22:02:03 DEBUG --- stderr --- 22:02:03 DEBUG 22:02:03 INFO 22:02:03 INFO [loop_until]: kubectl --namespace=xlou top node 22:02:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:02:03 INFO [loop_until]: OK (rc = 0) 22:02:03 DEBUG --- stdout --- 22:02:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5064Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4613m 15% 10733Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10273m 34% 10605Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16936m 56% 3051Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1961m 6% 4490Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6799m 22% 12766Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2952Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 9794m 32% 9815Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1936m 6% 3861Mi 3% gke-xlou-cdm-ds-0f601751-j488 4579m 15% 10269Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 13325m 44% 10313Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12134m 40% 7109Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6900m 43% 6012Mi 10% 22:02:03 DEBUG --- stderr --- 22:02:03 DEBUG 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713214983" 22:03:03 ERROR [http_cmd]: ERROR 22:03:03 ERROR --- status code --- 22:03:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:03:03 ERROR --- http response --- 22:03:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:03:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG { 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "data": { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "result": [ 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "2333270016" 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG --- http response --- 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG --- http response --- 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG { 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "data": { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "result": [ 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "instance": "10.142.0.107:9100" 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "instance": "10.142.0.43:9100" 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "11035.399999999998" 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO 22:03:03 INFO [loop_until]: kubectl --namespace=xlou top pods 22:03:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG --- http response --- 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG ----- output ----- 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG { 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG --- status code --- 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG "data": { 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG { 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG "result": [ 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG { 22:03:03 DEBUG "data": { 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG { 22:03:03 DEBUG { 22:03:03 DEBUG { 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG { 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG { 22:03:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:03:03 DEBUG "result": [ 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "data": { 22:03:03 DEBUG "data": { 22:03:03 DEBUG "data": { 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG }, 22:03:03 DEBUG { 22:03:03 DEBUG "data": { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "data": { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "result": [ 22:03:03 DEBUG "result": [ 22:03:03 DEBUG "result": [ 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG 22:03:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:03:03 DEBUG "result": [ 22:03:03 DEBUG { 22:03:03 DEBUG { 22:03:03 DEBUG { 22:03:03 DEBUG "result": [ 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "instance": "10.142.0.107:9100" 22:03:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:03:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:03:03 DEBUG 22:03:03 DEBUG "pod": "ds-idrepo-2" 22:03:03 DEBUG 22:03:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:03:03 DEBUG }, 22:03:03 DEBUG }, 22:03:03 DEBUG }, 22:03:03 DEBUG [...] 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214983" 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG [...] 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:03:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG "instance": "10.142.0.43:9100" 22:03:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:03:03 DEBUG "data": { 22:03:03 DEBUG "2294.3333333333335" 22:03:03 DEBUG }, 22:03:03 DEBUG "pod": "ds-idrepo-0" 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:03:03 DEBUG }, 22:03:03 DEBUG }, 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG ] 22:03:03 DEBUG "value": [ 22:03:03 DEBUG }, 22:03:03 DEBUG "0" 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "result": [ 22:03:03 DEBUG } 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG ] 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG { 22:03:03 DEBUG ] 22:03:03 DEBUG "557.6333333333333" 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG } 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "0.0003666666666669964" 22:03:03 DEBUG "383458.5" 22:03:03 DEBUG "metric": { 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG "0" 22:03:03 DEBUG ] 22:03:03 DEBUG "2295.5333333333333" 22:03:03 DEBUG ] 22:03:03 DEBUG ] 22:03:03 DEBUG "pod": "ds-idrepo-2" 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG }, 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG ] 22:03:03 DEBUG "value": [ 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG [...] 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG "pod": "ds-cts-2" 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG }, 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG "value": [ 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:03:03 DEBUG { 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG "0" 22:03:03 DEBUG { 22:03:03 DEBUG "data": { 22:03:03 DEBUG { 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:03:03 DEBUG ] 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG } 22:03:03 DEBUG "data": { 22:03:03 DEBUG "result": [ 22:03:03 DEBUG "data": { 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG { 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG ] 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG } 22:03:03 DEBUG "result": [ 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "result": [ 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG "data": { 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG } 22:03:03 DEBUG { 22:03:03 DEBUG "instance": "10.142.0.107:9100" 22:03:03 DEBUG { 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG "metric": { 22:03:03 DEBUG }, 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "result": [ 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:03:03 DEBUG "pod": "ds-idrepo-2" 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "pod": "ds-idrepo-2" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG { 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG }, 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713214983" 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713214983" 22:03:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713214983" 22:03:03 DEBUG }, 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG "metric": { 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG { 22:03:03 DEBUG 22:03:03 INFO [loop_until]: OK (rc = 0) 22:03:03 DEBUG "value": [ 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG "pod": "ds-idrepo-2" 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG 22:03:03 DEBUG --- stdout --- 22:03:03 DEBUG 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG }, 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG "data": { 22:03:03 DEBUG 22:03:03 DEBUG "data": { 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 10914m 8564Mi am-7d597cf7d6-8vr8f 9997m 8340Mi am-7d597cf7d6-9wz9h 13439m 9179Mi ds-cts-0 9522m 8431Mi ds-cts-1 6490m 8479Mi ds-cts-2 5621m 8662Mi ds-idrepo-0 12149m 4953Mi ds-idrepo-1 1916m 1814Mi ds-idrepo-2 1883m 1815Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 6m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6744m 2142Mi 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG --- stderr --- 22:03:03 DEBUG 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG "result": [ 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG [...] 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG "result": [ 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG { 22:03:03 DEBUG { 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG { 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "instance": "10.142.0.43:9100" 22:03:03 DEBUG 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "pod": "ds-cts-1" 22:03:03 DEBUG "data": { 22:03:03 DEBUG 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG "pod": "ds-cts-1" 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG [...] 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:03:03 DEBUG "data": { 22:03:03 DEBUG }, 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "pod": "ds-idrepo-0" 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG }, 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "pod": "ds-idrepo-0" 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 DEBUG 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "result": [ 22:03:03 DEBUG }, 22:03:03 DEBUG { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG { 22:03:03 DEBUG { 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG }, 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG { 22:03:03 DEBUG "result": [ 22:03:03 DEBUG 22:03:03 DEBUG { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "2963.633333333333" 22:03:03 DEBUG "value": [ 22:03:03 DEBUG { 22:03:03 DEBUG 22:03:03 DEBUG { 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG { 22:03:03 DEBUG 22:03:03 DEBUG "metric": { 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "data": { 22:03:03 DEBUG 22:03:03 DEBUG "data": { 22:03:03 DEBUG "data": { 22:03:03 DEBUG ] 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "status": "success", 22:03:03 INFO [http_cmd]: http status code OK 22:03:03 DEBUG "pod": "ds-idrepo-0" 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG "data": { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG 22:03:03 DEBUG "instance": "10.142.0.107:9100" 22:03:03 DEBUG "0" 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "result": [ 22:03:03 DEBUG 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG } 22:03:03 DEBUG "0" 22:03:03 DEBUG "data": { 22:03:03 DEBUG --- status code --- 22:03:03 DEBUG }, 22:03:03 DEBUG "data": { 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "pod": "ds-idrepo-2" 22:03:03 DEBUG [...] 22:03:03 DEBUG }, 22:03:03 DEBUG ] 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "result": [ 22:03:03 DEBUG ] 22:03:03 DEBUG ] 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG "result": [ 22:03:03 DEBUG { 22:03:03 DEBUG "metric": { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG } 22:03:03 DEBUG "result": [ 22:03:03 DEBUG [...] 22:03:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:03:03 DEBUG { 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG "result": [ 22:03:03 DEBUG --- http response --- 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "result": [ 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG "pod": "ds-cts-1" 22:03:03 DEBUG 22:03:03 DEBUG ] 22:03:03 DEBUG { 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG "metric": { 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG { 22:03:03 DEBUG "0" 22:03:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:03:03 DEBUG { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG } 22:03:03 DEBUG "metric": { 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "pod": "ds-cts-1" 22:03:03 DEBUG } 22:03:03 DEBUG "metric": { 22:03:03 DEBUG ] 22:03:03 DEBUG ----- output ----- 22:03:03 DEBUG "metric": { 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG } 22:03:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG } 22:03:03 DEBUG "instance": "10.142.0.107:9100" 22:03:03 DEBUG } 22:03:03 DEBUG { 22:03:03 DEBUG "pod": "ds-idrepo-2" 22:03:03 DEBUG 22:03:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG }, 22:03:03 DEBUG ] 22:03:03 DEBUG "status": "success", 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG } 22:03:03 DEBUG "data": { 22:03:03 DEBUG "value": [ 22:03:03 DEBUG [...] 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG } 22:03:03 DEBUG "resultType": "vector", 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG "result": [ 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "0" 22:03:03 DEBUG 22:03:03 DEBUG "instance": "10.142.0.43:9100" 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG { 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG ] 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG "0" 22:03:03 DEBUG ] 22:03:03 DEBUG [...] 22:03:03 DEBUG 22:03:03 DEBUG [...] 22:03:03 DEBUG "metric": { 22:03:03 DEBUG [...] 22:03:03 DEBUG "pod": "ds-idrepo-0" 22:03:03 DEBUG } 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG } 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG 22:03:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG ] 22:03:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:03:03 DEBUG ] 22:03:03 DEBUG 22:03:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG } 22:03:03 DEBUG }, 22:03:03 DEBUG "0" 22:03:03 DEBUG }, 22:03:03 DEBUG } 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG } 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "0" 22:03:03 DEBUG ] 22:03:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "instance": "10.142.0.43:9100" 22:03:03 DEBUG 22:03:03 DEBUG "pod": "ds-idrepo-0" 22:03:03 DEBUG "0" 22:03:03 DEBUG "value": [ 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG }, 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG }, 22:03:03 DEBUG 22:03:03 DEBUG }, 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG "value": [ 22:03:03 DEBUG "0.078" 22:03:03 DEBUG "value": [ 22:03:03 DEBUG 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG "2298.4666666666667" 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG ] 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG [...] 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG ] 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG "0.00001143451456557749" 22:03:03 DEBUG } 22:03:03 DEBUG "13855167488" 22:03:03 DEBUG 22:03:03 DEBUG "0" 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG ] 22:03:03 DEBUG ] 22:03:03 DEBUG 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG }, 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 DEBUG "value": [ 22:03:03 DEBUG } 22:03:03 DEBUG 1713214983, 22:03:03 DEBUG "6.761865700108222" 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG ] 22:03:03 DEBUG } 22:03:03 DEBUG } 22:03:03 INFO 22:03:03 INFO [loop_until]: kubectl --namespace=xlou top node 22:03:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:03:03 INFO [loop_until]: OK (rc = 0) 22:03:03 DEBUG --- stdout --- 22:03:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5077Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6410m 21% 10684Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10910m 36% 10620Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17487m 58% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1941m 6% 4647Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9037m 30% 12738Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2965Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10361m 34% 9832Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1999m 6% 4061Mi 3% gke-xlou-cdm-ds-0f601751-j488 6439m 21% 10233Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2452Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 14817m 49% 10794Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12286m 41% 7120Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6884m 43% 6016Mi 10% 22:03:03 DEBUG --- stderr --- 22:03:03 DEBUG 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215043" 22:04:03 ERROR [http_cmd]: ERROR 22:04:03 ERROR --- status code --- 22:04:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:04:03 ERROR --- http response --- 22:04:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:04:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "data": { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "result": [ 22:04:03 DEBUG { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG "2404691968" 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215043" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG --- http response --- 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG --- http response --- 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG { 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "data": { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "result": [ 22:04:03 DEBUG { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG "2268.3333333333335" 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "data": { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "result": [ 22:04:03 DEBUG { 22:04:03 DEBUG "metric": { 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG "instance": "10.142.0.107:9100" 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG }, 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "value": [ 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG 22:04:03 DEBUG "data": { 22:04:03 DEBUG "instance": "10.142.0.43:9100" 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG }, 22:04:03 DEBUG "result": [ 22:04:03 DEBUG "value": [ 22:04:03 DEBUG { 22:04:03 INFO 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG "metric": { 22:04:03 INFO [loop_until]: kubectl --namespace=xlou top pods 22:04:03 DEBUG "11590.999999999996" 22:04:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:04:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG ] 22:04:03 DEBUG }, 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG } 22:04:03 DEBUG "value": [ 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG ] 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 DEBUG 22:04:03 DEBUG --- http response --- 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215043" 22:04:03 DEBUG --- status code --- 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG [...] 22:04:03 DEBUG { 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG "status": "success", 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG "data": { 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG { 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG { 22:04:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:04:03 DEBUG }, 22:04:03 DEBUG { 22:04:03 DEBUG "result": [ 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "data": { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG "data": { 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG "data": { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "result": [ 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "0" 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG "resultType": "vector", 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG "instance": "10.142.0.107:9100" 22:04:03 DEBUG { 22:04:03 DEBUG { 22:04:03 DEBUG "result": [ 22:04:03 DEBUG ] 22:04:03 DEBUG { 22:04:03 DEBUG "result": [ 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG }, 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG { 22:04:03 DEBUG } 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG { 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:04:03 DEBUG "data": { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG ] 22:04:03 DEBUG "data": { 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG "metric": { 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:04:03 DEBUG } 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "result": [ 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "result": [ 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG }, 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "value": [ 22:04:03 DEBUG { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG "metric": { 22:04:03 DEBUG 22:04:03 DEBUG "metric": { 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG "data": { 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:04:03 DEBUG 22:04:03 DEBUG "pod": "ds-idrepo-2" 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG }, 22:04:03 DEBUG [...] 22:04:03 DEBUG }, 22:04:03 DEBUG { 22:04:03 DEBUG 22:04:03 DEBUG "data": { 22:04:03 DEBUG "result": [ 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG [...] 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG { 22:04:03 DEBUG "instance": "10.142.0.43:9100" 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "data": { 22:04:03 DEBUG 22:04:03 DEBUG "result": [ 22:04:03 DEBUG "metric": { 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:04:03 DEBUG 22:04:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:04:03 DEBUG 22:04:03 DEBUG "result": [ 22:04:03 DEBUG 22:04:03 DEBUG "metric": { 22:04:03 DEBUG }, 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG }, 22:04:03 DEBUG [...] 22:04:03 DEBUG }, 22:04:03 DEBUG [...] 22:04:03 DEBUG { 22:04:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:04:03 DEBUG "instance": "10.142.0.107:9100" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "0.0003666666666669964" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG "metric": { 22:04:03 DEBUG }, 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG "2268.866666666667" 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG 22:04:03 DEBUG "pod": "ds-idrepo-2" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG "2300.2" 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG "pod": "ds-idrepo-0" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "412598.5" 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG } 22:04:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG }, 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG } 22:04:03 DEBUG "0" 22:04:03 DEBUG [...] 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:04:03 DEBUG "0.078" 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG "instance": "10.142.0.43:9100" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG "pod": "ds-cts-2" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "0" 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG }, 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "3235.0333333333333" 22:04:03 DEBUG } 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG ] 22:04:03 DEBUG ] 22:04:03 DEBUG "0" 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG { 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "data": { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "result": [ 22:04:03 DEBUG { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "pod": "ds-idrepo-2" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "pod": "ds-idrepo-0" 22:04:03 DEBUG }, 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG "0" 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215043" 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215043" 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215043" 22:04:03 DEBUG --- http response --- 22:04:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215043" 22:04:03 INFO [loop_until]: OK (rc = 0) 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG --- stdout --- 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12438m 8659Mi am-7d597cf7d6-8vr8f 12104m 8360Mi am-7d597cf7d6-9wz9h 15998m 9226Mi ds-cts-0 14932m 8435Mi ds-cts-1 10738m 8460Mi ds-cts-2 10417m 8655Mi ds-idrepo-0 13603m 4953Mi ds-idrepo-1 2004m 1967Mi ds-idrepo-2 2043m 1929Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6702m 2239Mi 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG { 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG --- stderr --- 22:04:03 DEBUG { 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG "status": "success", 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG { 22:04:03 DEBUG "data": { 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG "data": { 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "data": { 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG "result": [ 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG { 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "result": [ 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG { 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG { 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG "result": [ 22:04:03 DEBUG { 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:04:03 DEBUG "metric": { 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "data": { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "metric": { 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG "pod": "ds-idrepo-2" 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:04:03 DEBUG { 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "pod": "ds-idrepo-2" 22:04:03 DEBUG { 22:04:03 DEBUG }, 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG "result": [ 22:04:03 DEBUG }, 22:04:03 INFO [http_cmd]: http status code OK 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG }, 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "value": [ 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG { 22:04:03 DEBUG "value": [ 22:04:03 DEBUG --- status code --- 22:04:03 DEBUG "data": { 22:04:03 DEBUG "value": [ 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG "data": { 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:04:03 DEBUG { 22:04:03 DEBUG "metric": { 22:04:03 DEBUG 22:04:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:04:03 DEBUG 22:04:03 DEBUG --- http response --- 22:04:03 DEBUG "result": [ 22:04:03 DEBUG 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "result": [ 22:04:03 DEBUG 22:04:03 DEBUG "data": { 22:04:03 DEBUG { 22:04:03 DEBUG "data": { 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:04:03 DEBUG 22:04:03 DEBUG "data": { 22:04:03 DEBUG { 22:04:03 DEBUG [...] 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "value": [ 22:04:03 DEBUG [...] 22:04:03 DEBUG "metric": { 22:04:03 DEBUG ----- output ----- 22:04:03 DEBUG [...] 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "metric": { 22:04:03 DEBUG 22:04:03 DEBUG "result": [ 22:04:03 DEBUG "data": { 22:04:03 DEBUG "result": [ 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "pod": "ds-idrepo-2" 22:04:03 DEBUG { 22:04:03 DEBUG 22:04:03 DEBUG "result": [ 22:04:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG { 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG "status": "success", 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "result": [ 22:04:03 DEBUG "metric": { 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "data": { 22:04:03 DEBUG 22:04:03 DEBUG "metric": { 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "pod": "ds-idrepo-0" 22:04:03 DEBUG "instance": "10.142.0.107:9100" 22:04:03 DEBUG { 22:04:03 DEBUG "instance": "10.142.0.107:9100" 22:04:03 DEBUG [...] 22:04:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:04:03 DEBUG 22:04:03 DEBUG "resultType": "vector", 22:04:03 DEBUG "pod": "ds-idrepo-0" 22:04:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG }, 22:04:03 DEBUG "metric": { 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG "result": [ 22:04:03 DEBUG }, 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG { 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG [...] 22:04:03 DEBUG "metric": { 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG "0" 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:04:03 DEBUG "0" 22:04:03 DEBUG 22:04:03 DEBUG "pod": "ds-idrepo-2" 22:04:03 DEBUG "0" 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG "value": [ 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG } 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG ] 22:04:03 DEBUG "pod": "ds-idrepo-0" 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG [...] 22:04:03 DEBUG 22:04:03 DEBUG "2033154.2435710027" 22:04:03 DEBUG } 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG "value": [ 22:04:03 DEBUG 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "instance": "10.142.0.43:9100" 22:04:03 DEBUG 22:04:03 DEBUG "instance": "10.142.0.43:9100" 22:04:03 DEBUG } 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG [...] 22:04:03 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG }, 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG ] 22:04:03 DEBUG "0" 22:04:03 DEBUG 22:04:03 DEBUG }, 22:04:03 DEBUG "2272.233333333333" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG 22:04:03 DEBUG "value": [ 22:04:03 DEBUG ] 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG }, 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG } 22:04:03 DEBUG "0.0006666666666666673" 22:04:03 DEBUG "value": [ 22:04:03 DEBUG "13855935488" 22:04:03 DEBUG ] 22:04:03 DEBUG "pod": "ds-idrepo-0" 22:04:03 DEBUG "0.000660220485986851" 22:04:03 DEBUG ] 22:04:03 DEBUG ] 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG }, 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG "6.72753007030144" 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG "value": [ 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG ] 22:04:03 DEBUG ] 22:04:03 DEBUG 1713215043, 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG "0" 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 DEBUG ] 22:04:03 DEBUG } 22:04:03 DEBUG } 22:04:03 INFO 22:04:03 INFO [loop_until]: kubectl --namespace=xlou top node 22:04:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:04:03 INFO [loop_until]: OK (rc = 0) 22:04:03 DEBUG --- stdout --- 22:04:03 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10330m 34% 10667Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13137m 44% 10733Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17473m 58% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2073m 6% 4762Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15730m 52% 12713Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2964Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12796m 42% 9854Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 2167m 7% 4184Mi 3% gke-xlou-cdm-ds-0f601751-j488 10589m 35% 10227Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 59m 0% 2453Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16861m 56% 10857Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13675m 45% 7118Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6905m 43% 6113Mi 10% 22:04:03 DEBUG --- stderr --- 22:04:03 DEBUG 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215103" 22:05:03 ERROR [http_cmd]: ERROR 22:05:03 ERROR --- status code --- 22:05:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:05:03 ERROR --- http response --- 22:05:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:05:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG --- http response --- 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "2444763136" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "0.078" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "2304.4333333333334" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:05:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "2286.733333333333" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "instance": "10.142.0.107:9100" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "instance": "10.142.0.43:9100" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "11402.266666666665" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "ds-idrepo-2" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "pod": "ds-idrepo-0" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "0" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG --- http response --- 22:05:03 INFO 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [loop_until]: kubectl --namespace=xlou top pods 22:05:03 DEBUG { 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:05:03 DEBUG "status": "success", 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215103" 22:05:03 DEBUG "data": { 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG "resultType": "vector", 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215103" 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG "result": [ 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG { 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "metric": { 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG }, 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG { 22:05:03 DEBUG { 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG "value": [ 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "data": { 22:05:03 DEBUG "data": { 22:05:03 DEBUG { 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG { 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG { 22:05:03 DEBUG { 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG "data": { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG "status": "success", 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG "result": [ 22:05:03 DEBUG "data": { 22:05:03 DEBUG "data": { 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:05:03 DEBUG "data": { 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG "data": { 22:05:03 DEBUG "data": { 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG [...] 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG { 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "result": [ 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG "result": [ 22:05:03 DEBUG "result": [ 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:05:03 DEBUG "instance": "10.142.0.107:9100" 22:05:03 DEBUG { 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG { 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG "pod": "ds-idrepo-2" 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "data": { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "data": { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "pod": "ds-idrepo-2" 22:05:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:05:03 DEBUG "instance": "10.142.0.107:9100" 22:05:03 DEBUG }, 22:05:03 DEBUG { 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG "result": [ 22:05:03 DEBUG }, 22:05:03 DEBUG "result": [ 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG { 22:05:03 DEBUG "value": [ 22:05:03 DEBUG { 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "data": { 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "metric": { 22:05:03 DEBUG 22:05:03 DEBUG "metric": { 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "0" 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "pod": "ds-idrepo-2" 22:05:03 DEBUG 22:05:03 DEBUG "instance": "10.142.0.107:9100" 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG ] 22:05:03 DEBUG "result": [ 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG [...] 22:05:03 DEBUG "value": [ 22:05:03 DEBUG [...] 22:05:03 DEBUG "value": [ 22:05:03 DEBUG [...] 22:05:03 DEBUG [...] 22:05:03 DEBUG ] 22:05:03 DEBUG "metric": { 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:05:03 DEBUG 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG "instance": "10.142.0.43:9100" 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215103" 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG }, 22:05:03 DEBUG "pod": "ds-idrepo-0" 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG [...] 22:05:03 DEBUG 22:05:03 DEBUG "value": [ 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "pod": "ds-idrepo-0" 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG "instance": "10.142.0.43:9100" 22:05:03 DEBUG 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "2287.1" 22:05:03 DEBUG "0.00033333333333350387" 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG "0" 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "pod": "ds-cts-2" 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "instance": "10.142.0.43:9100" 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG [...] 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG "0" 22:05:03 DEBUG "2285.866666666667" 22:05:03 DEBUG }, 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG }, 22:05:03 DEBUG "3014.6" 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "412087.5" 22:05:03 DEBUG "10604338.463352792" 22:05:03 DEBUG "value": [ 22:05:03 DEBUG ] 22:05:03 DEBUG 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "0" 22:05:03 DEBUG ] 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG } 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG "13856439296" 22:05:03 DEBUG ] 22:05:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG }, 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG "value": [ 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215103" 22:05:03 DEBUG ] 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG "6.7074929029734855" 22:05:03 DEBUG } 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG ] 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG } 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG ] 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG } 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG } 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG { 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:05:03 DEBUG "data": { 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:05:03 DEBUG ----- output ----- 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG ----- output ----- 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG { 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG { 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG "data": { 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "data": { 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "instance": "10.142.0.107:9100" 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "result": [ 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:05:03 DEBUG }, 22:05:03 INFO [loop_until]: OK (rc = 0) 22:05:03 DEBUG ----- output ----- 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG "result": [ 22:05:03 DEBUG { 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG "value": [ 22:05:03 DEBUG --- stdout --- 22:05:03 DEBUG { 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG { 22:05:03 DEBUG "metric": { 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12520m 8612Mi am-7d597cf7d6-8vr8f 11729m 9055Mi am-7d597cf7d6-9wz9h 15681m 9217Mi ds-cts-0 14880m 8437Mi ds-cts-1 10870m 8468Mi ds-cts-2 10278m 8649Mi ds-idrepo-0 13510m 4953Mi ds-idrepo-1 2026m 2049Mi ds-idrepo-2 2062m 2060Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6696m 2241Mi 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "ds-idrepo-2" 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG 22:05:03 DEBUG "data": { 22:05:03 DEBUG --- stderr --- 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:05:03 DEBUG }, 22:05:03 DEBUG "data": { 22:05:03 DEBUG 22:05:03 INFO [http_cmd]: http status code OK 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG [...] 22:05:03 DEBUG --- status code --- 22:05:03 DEBUG "result": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG 22:05:03 DEBUG "result": [ 22:05:03 DEBUG 22:05:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG --- http response --- 22:05:03 DEBUG "metric": { 22:05:03 DEBUG 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG 22:05:03 DEBUG "metric": { 22:05:03 DEBUG 22:05:03 DEBUG "pod": "ds-idrepo-2" 22:05:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:05:03 DEBUG 22:05:03 DEBUG "data": { 22:05:03 DEBUG [...] 22:05:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:05:03 DEBUG "instance": "10.142.0.43:9100" 22:05:03 DEBUG }, 22:05:03 DEBUG ----- output ----- 22:05:03 DEBUG [...] 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG "result": [ 22:05:03 DEBUG 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG "status": "success", 22:05:03 DEBUG 22:05:03 DEBUG { 22:05:03 DEBUG 22:05:03 DEBUG 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG 22:05:03 DEBUG "data": { 22:05:03 DEBUG 22:05:03 DEBUG "metric": { 22:05:03 DEBUG "pod": "ds-idrepo-0" 22:05:03 DEBUG 22:05:03 DEBUG "0.00033333333333333365" 22:05:03 DEBUG 22:05:03 DEBUG "resultType": "vector", 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:05:03 DEBUG }, 22:05:03 DEBUG 22:05:03 DEBUG ] 22:05:03 DEBUG [...] 22:05:03 DEBUG "result": [ 22:05:03 DEBUG }, 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG [...] 22:05:03 DEBUG } 22:05:03 DEBUG 22:05:03 DEBUG { 22:05:03 DEBUG "value": [ 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG 22:05:03 DEBUG ] 22:05:03 DEBUG 22:05:03 DEBUG "metric": { 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG 22:05:03 DEBUG "0" 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG 22:05:03 DEBUG "pod": "ds-idrepo-2" 22:05:03 DEBUG "0" 22:05:03 DEBUG 22:05:03 DEBUG ] 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG "pod": "ds-idrepo-0" 22:05:03 DEBUG }, 22:05:03 DEBUG ] 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG } 22:05:03 DEBUG [...] 22:05:03 DEBUG ] 22:05:03 DEBUG }, 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 22:05:03 DEBUG ] 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG "value": [ 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "0" 22:05:03 DEBUG 22:05:03 DEBUG } 22:05:03 DEBUG 22:05:03 DEBUG "35745.50386225603" 22:05:03 DEBUG ] 22:05:03 DEBUG [...] 22:05:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG 22:05:03 DEBUG }, 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG 22:05:03 DEBUG "value": [ 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG "pod": "ds-idrepo-0" 22:05:03 DEBUG "15.821557108320492" 22:05:03 DEBUG } 22:05:03 DEBUG }, 22:05:03 DEBUG ] 22:05:03 DEBUG "value": [ 22:05:03 DEBUG } 22:05:03 DEBUG 1713215103, 22:05:03 DEBUG "0" 22:05:03 DEBUG ] 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:03 DEBUG ] 22:05:03 DEBUG } 22:05:03 DEBUG } 22:05:04 INFO 22:05:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:05:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:05:04 INFO [loop_until]: OK (rc = 0) 22:05:04 DEBUG --- stdout --- 22:05:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10706m 35% 10677Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13235m 44% 10694Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17574m 58% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2100m 7% 4844Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15968m 53% 12726Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2952Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12716m 42% 10548Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2149m 7% 4273Mi 3% gke-xlou-cdm-ds-0f601751-j488 10597m 35% 10221Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16965m 56% 10854Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13783m 46% 7117Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6861m 43% 6113Mi 10% 22:05:04 DEBUG --- stderr --- 22:05:04 DEBUG 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215163" 22:06:03 ERROR [http_cmd]: ERROR 22:06:03 ERROR --- status code --- 22:06:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:06:03 ERROR --- http response --- 22:06:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:06:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "2452164608" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "2278.4333333333334" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "2294.9333333333334" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "0.078" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "ds-idrepo-2" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "ds-idrepo-0" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "0" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "instance": "10.142.0.107:9100" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "instance": "10.142.0.43:9100" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "11554.399999999998" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "instance": "10.142.0.107:9100" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "instance": "10.142.0.43:9100" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "3169" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "ds-idrepo-2" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "ds-idrepo-0" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "0" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "0" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "ds-idrepo-2" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "ds-cts-2" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "0" 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG ] 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "2279.0333333333333" 22:06:03 DEBUG ] 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG } 22:06:03 DEBUG ----- output ----- 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 DEBUG ] 22:06:03 DEBUG { 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215163" 22:06:03 DEBUG } 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG } 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG "data": { 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG "result": [ 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG { 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG "metric": { 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG { 22:06:03 DEBUG "instance": "10.142.0.107:9100" 22:06:03 INFO 22:06:03 DEBUG { 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG }, 22:06:03 INFO [loop_until]: kubectl --namespace=xlou top pods 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG "data": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:06:03 DEBUG 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "result": [ 22:06:03 DEBUG 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 DEBUG [...] 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG { 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 DEBUG "metric": { 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215163" 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG "metric": { 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG "pod": "ds-idrepo-2" 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG }, 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG }, 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG "value": [ 22:06:03 DEBUG --- http response --- 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG "instance": "10.142.0.43:9100" 22:06:03 DEBUG "value": [ 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:06:03 DEBUG }, 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG "value": [ 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG 22:06:03 DEBUG { 22:06:03 DEBUG 22:06:03 DEBUG { 22:06:03 DEBUG { 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG { 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG { 22:06:03 DEBUG 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG [...] 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:06:03 DEBUG "0.00030000000000001136" 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG [...] 22:06:03 DEBUG "data": { 22:06:03 DEBUG 22:06:03 DEBUG "data": { 22:06:03 DEBUG "data": { 22:06:03 DEBUG "data": { 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG ] 22:06:03 DEBUG "data": { 22:06:03 DEBUG 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG { 22:06:03 DEBUG } 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG 22:06:03 DEBUG "result": [ 22:06:03 DEBUG 22:06:03 DEBUG "result": [ 22:06:03 DEBUG "result": [ 22:06:03 DEBUG "result": [ 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG ] 22:06:03 DEBUG "result": [ 22:06:03 DEBUG 22:06:03 DEBUG { 22:06:03 DEBUG "pod": "ds-idrepo-0" 22:06:03 DEBUG { 22:06:03 DEBUG { 22:06:03 DEBUG { 22:06:03 DEBUG "data": { 22:06:03 DEBUG } 22:06:03 DEBUG { 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG "metric": { 22:06:03 DEBUG }, 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG } 22:06:03 DEBUG "metric": { 22:06:03 DEBUG }, 22:06:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:06:03 DEBUG "instance": "10.142.0.107:9100" 22:06:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:06:03 DEBUG "result": [ 22:06:03 DEBUG { 22:06:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:06:03 DEBUG }, 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG }, 22:06:03 DEBUG }, 22:06:03 DEBUG }, 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "metric": { 22:06:03 DEBUG }, 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "0" 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "value": [ 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "pod": "ds-idrepo-2" 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG ] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG "2282.0666666666666" 22:06:03 DEBUG }, 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG } 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG ] 22:06:03 DEBUG "value": [ 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG ] 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG } 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG [...] 22:06:03 DEBUG } 22:06:03 DEBUG [...] 22:06:03 DEBUG [...] 22:06:03 DEBUG [...] 22:06:03 DEBUG ] 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG } 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG } 22:06:03 DEBUG { 22:06:03 DEBUG [...] 22:06:03 DEBUG [...] 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG } 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG "data": { 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG "instance": "10.142.0.43:9100" 22:06:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG 22:06:03 DEBUG 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG }, 22:06:03 DEBUG }, 22:06:03 DEBUG }, 22:06:03 DEBUG }, 22:06:03 DEBUG "result": [ 22:06:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:06:03 DEBUG "pod": "ds-idrepo-0" 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:06:03 DEBUG "value": [ 22:06:03 INFO [http_cmd]: http status code OK 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "value": [ 22:06:03 DEBUG { 22:06:03 DEBUG }, 22:06:03 DEBUG }, 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG --- status code --- 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "metric": { 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "value": [ 22:06:03 DEBUG { 22:06:03 DEBUG { 22:06:03 DEBUG { 22:06:03 DEBUG "411594" 22:06:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:06:03 DEBUG "11027663.34752903" 22:06:03 DEBUG "13857002496" 22:06:03 DEBUG "15.84380973681735" 22:06:03 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG "6.714826492716809" 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "status": "success", 22:06:03 DEBUG "data": { 22:06:03 DEBUG --- http response --- 22:06:03 DEBUG ] 22:06:03 DEBUG ] 22:06:03 DEBUG ] 22:06:03 DEBUG }, 22:06:03 DEBUG 1713215163, 22:06:03 DEBUG ] 22:06:03 DEBUG "data": { 22:06:03 DEBUG "data": { 22:06:03 DEBUG ] 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 DEBUG } 22:06:03 DEBUG "value": [ 22:06:03 DEBUG "0" 22:06:03 DEBUG } 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG "resultType": "vector", 22:06:03 DEBUG } 22:06:03 DEBUG "result": [ 22:06:03 DEBUG ----- output ----- 22:06:03 DEBUG ] 22:06:03 DEBUG ] 22:06:03 DEBUG ] 22:06:03 DEBUG 22:06:03 DEBUG ] 22:06:03 DEBUG ] 22:06:03 DEBUG "result": [ 22:06:03 DEBUG "result": [ 22:06:04 DEBUG ] 22:06:04 DEBUG { 22:06:04 DEBUG { 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG { 22:06:04 DEBUG { 22:06:04 DEBUG } 22:06:04 DEBUG "metric": { 22:06:04 DEBUG "status": "success", 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG 22:06:04 DEBUG ] 22:06:04 DEBUG } 22:06:04 DEBUG "metric": { 22:06:04 DEBUG "metric": { 22:06:04 DEBUG } 22:06:04 DEBUG "pod": "ds-idrepo-2" 22:06:04 DEBUG "data": { 22:06:04 DEBUG [...] 22:06:04 DEBUG } 22:06:04 DEBUG "pod": "ds-idrepo-2" 22:06:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:06:04 DEBUG }, 22:06:04 DEBUG "resultType": "vector", 22:06:04 DEBUG 22:06:04 DEBUG } 22:06:04 DEBUG }, 22:06:04 DEBUG }, 22:06:04 DEBUG "value": [ 22:06:04 DEBUG "result": [ 22:06:04 DEBUG 22:06:04 DEBUG "value": [ 22:06:04 DEBUG "value": [ 22:06:04 DEBUG 22:06:04 DEBUG { 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG "metric": { 22:06:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG "instance": "10.142.0.107:9100" 22:06:04 DEBUG }, 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG [...] 22:06:04 DEBUG }, 22:06:04 DEBUG "value": [ 22:06:04 DEBUG [...] 22:06:04 DEBUG [...] 22:06:04 DEBUG 22:06:04 DEBUG "value": [ 22:06:04 DEBUG 1713215163, 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG "0" 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG ] 22:06:04 DEBUG 22:06:04 DEBUG 22:06:04 DEBUG "pod": "ds-idrepo-0" 22:06:04 DEBUG 22:06:04 DEBUG } 22:06:04 DEBUG "pod": "ds-idrepo-0" 22:06:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:06:04 DEBUG }, 22:06:04 DEBUG [...] 22:06:04 DEBUG ] 22:06:04 DEBUG }, 22:06:04 DEBUG }, 22:06:04 DEBUG "value": [ 22:06:04 DEBUG 22:06:04 DEBUG } 22:06:04 DEBUG "value": [ 22:06:04 DEBUG "value": [ 22:06:04 DEBUG 1713215163, 22:06:04 DEBUG 22:06:04 DEBUG } 22:06:04 DEBUG 1713215163, 22:06:04 DEBUG 1713215163, 22:06:04 DEBUG "0" 22:06:04 DEBUG 22:06:04 DEBUG "0" 22:06:04 DEBUG "164648.18162321148" 22:06:04 DEBUG ] 22:06:04 DEBUG "instance": "10.142.0.43:9100" 22:06:04 DEBUG ] 22:06:04 DEBUG ] 22:06:04 DEBUG } 22:06:04 DEBUG }, 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG ] 22:06:04 DEBUG "value": [ 22:06:04 DEBUG ] 22:06:04 DEBUG ] 22:06:04 DEBUG } 22:06:04 DEBUG 1713215163, 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG "0" 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 DEBUG ] 22:06:04 DEBUG } 22:06:04 DEBUG ] 22:06:04 DEBUG } 22:06:04 DEBUG } 22:06:04 INFO [loop_until]: OK (rc = 0) 22:06:04 DEBUG --- stdout --- 22:06:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12612m 8611Mi am-7d597cf7d6-8vr8f 11841m 9066Mi am-7d597cf7d6-9wz9h 15766m 9221Mi ds-cts-0 15062m 8425Mi ds-cts-1 10865m 8468Mi ds-cts-2 10810m 8652Mi ds-idrepo-0 13714m 4953Mi ds-idrepo-1 2015m 2123Mi ds-idrepo-2 2007m 2104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 1026Mi idm-b85f9d978-rl76f 5m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6555m 2243Mi 22:06:04 DEBUG --- stderr --- 22:06:04 DEBUG 22:06:04 INFO 22:06:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:06:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:06:04 INFO [loop_until]: OK (rc = 0) 22:06:04 DEBUG --- stdout --- 22:06:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5091Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10878m 36% 10679Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13094m 43% 10686Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17512m 58% 3047Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2085m 6% 4932Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16345m 54% 12734Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 68m 0% 2963Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12597m 42% 10554Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2111m 7% 4333Mi 3% gke-xlou-cdm-ds-0f601751-j488 10674m 35% 10229Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2455Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16879m 56% 10854Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13737m 46% 7111Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6839m 43% 6117Mi 10% 22:06:04 DEBUG --- stderr --- 22:06:04 DEBUG 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215223" 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215223" 22:07:03 ERROR [http_cmd]: ERROR 22:07:03 ERROR --- status code --- 22:07:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:07:03 ERROR --- http response --- 22:07:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:07:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215223" 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215223" 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1713215223, 22:07:03 DEBUG "2269.0333333333333" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1713215223, 22:07:03 DEBUG "2285.233333333333" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1713215223, 22:07:03 DEBUG "2450837504" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215223" 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "instance": "10.142.0.107:9100" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "instance": "10.142.0.43:9100" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1713215223, 22:07:03 DEBUG "11278.999999999998" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215223" 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215223" 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215223" 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1713215223, 22:07:03 DEBUG "0.078" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "pod": "ds-idrepo-2" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "pod": "ds-idrepo-0" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1713215223, 22:07:03 DEBUG "0" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215223" 22:07:03 INFO [http_cmd]: http status code OK 22:07:03 DEBUG --- status code --- 22:07:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:03 DEBUG --- http response --- 22:07:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:03 DEBUG ----- output ----- 22:07:03 DEBUG { 22:07:03 DEBUG "status": "success", 22:07:03 DEBUG "data": { 22:07:03 DEBUG "resultType": "vector", 22:07:03 DEBUG "result": [ 22:07:03 DEBUG { 22:07:03 DEBUG "metric": { 22:07:03 DEBUG "instance": "10.142.0.107:9100" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG [...] 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG 22:07:03 DEBUG "instance": "10.142.0.43:9100" 22:07:03 DEBUG }, 22:07:03 DEBUG "value": [ 22:07:03 DEBUG 1713215223, 22:07:03 DEBUG "2988.866666666667" 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG ] 22:07:03 DEBUG } 22:07:03 DEBUG } 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "pod": "ds-idrepo-2" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "pod": "ds-idrepo-0" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "0" 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG --- http response --- 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:07:04 DEBUG "pod": "ds-idrepo-2" 22:07:04 DEBUG }, 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:07:04 DEBUG "pod": "ds-cts-2" 22:07:04 DEBUG }, 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "0" 22:07:04 DEBUG "0" 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "2285.1666666666665" 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "411816.5" 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "pod": "ds-idrepo-2" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "pod": "ds-idrepo-0" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "0" 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "instance": "10.142.0.107:9100" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "instance": "10.142.0.43:9100" 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "0.00043333333333350765" 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215224" 22:07:04 DEBUG --- status code --- 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 INFO 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG --- status code --- 22:07:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG ----- output ----- 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG { 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG --- status code --- 22:07:04 INFO [http_cmd]: http status code OK 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- status code --- 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG "data": { 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG { 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG --- http response --- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG "result": [ 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:04 DEBUG { 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:07:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG "metric": { 22:07:04 DEBUG { 22:07:04 DEBUG { 22:07:04 DEBUG { 22:07:04 DEBUG "data": { 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG { 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG ----- output ----- 22:07:04 DEBUG { 22:07:04 DEBUG "instance": "10.142.0.107:9100" 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG { 22:07:04 DEBUG "data": { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG { 22:07:04 DEBUG { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG }, 22:07:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:07:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:07:04 DEBUG "data": { 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "status": "success", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "value": [ 22:07:04 DEBUG }, 22:07:04 DEBUG }, 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG { 22:07:04 DEBUG "data": { 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "data": { 22:07:04 DEBUG "data": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG { 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "resultType": "vector", 22:07:04 DEBUG "result": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG { 22:07:04 DEBUG "pod": "ds-idrepo-2" 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "metric": { 22:07:04 DEBUG { 22:07:04 DEBUG "result": [ 22:07:04 DEBUG "result": [ 22:07:04 DEBUG { 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "metric": { 22:07:04 DEBUG }, 22:07:04 DEBUG { 22:07:04 DEBUG "pod": "ds-idrepo-2" 22:07:04 DEBUG "metric": { 22:07:04 DEBUG { 22:07:04 DEBUG { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "metric": { 22:07:04 DEBUG }, 22:07:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "metric": { 22:07:04 DEBUG "instance": "10.142.0.107:9100" 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG [...] 22:07:04 DEBUG }, 22:07:04 DEBUG 22:07:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:07:04 DEBUG "value": [ 22:07:04 DEBUG }, 22:07:04 DEBUG "pod": "ds-idrepo-2" 22:07:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:07:04 DEBUG }, 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG }, 22:07:04 DEBUG 22:07:04 DEBUG "value": [ 22:07:04 DEBUG }, 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG "instance": "10.142.0.43:9100" 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG }, 22:07:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:07:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "value": [ 22:07:04 DEBUG }, 22:07:04 DEBUG }, 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG [...] 22:07:04 DEBUG [...] 22:07:04 DEBUG 22:07:04 DEBUG "13857620992" 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG 22:07:04 DEBUG "pod": "ds-idrepo-0" 22:07:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG ] 22:07:04 DEBUG "2284.4333333333334" 22:07:04 DEBUG "0" 22:07:04 DEBUG [...] 22:07:04 DEBUG }, 22:07:04 DEBUG }, 22:07:04 DEBUG "pod": "ds-idrepo-0" 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "value": [ 22:07:04 DEBUG }, 22:07:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:07:04 DEBUG 22:07:04 DEBUG 22:07:04 DEBUG "instance": "10.142.0.43:9100" 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG }, 22:07:04 DEBUG "pod": "ds-idrepo-0" 22:07:04 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 22:07:04 DEBUG }, 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG 22:07:04 DEBUG "0" 22:07:04 DEBUG ] 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG }, 22:07:04 DEBUG }, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 22:07:04 DEBUG }, 22:07:04 DEBUG "6.710241000355494" 22:07:04 DEBUG "0" 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "value": [ 22:07:04 DEBUG "value": [ 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG "value": [ 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG "0" 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG "0.00033333333333333365" 22:07:04 DEBUG ] 22:07:04 DEBUG 1713215224, 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG "0" 22:07:04 DEBUG "0" 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG "0" 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG ] 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 DEBUG } 22:07:04 INFO [loop_until]: OK (rc = 0) 22:07:04 DEBUG --- stdout --- 22:07:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12617m 8612Mi am-7d597cf7d6-8vr8f 11621m 9064Mi am-7d597cf7d6-9wz9h 15344m 9217Mi ds-cts-0 15282m 8421Mi ds-cts-1 11320m 8495Mi ds-cts-2 10531m 8680Mi ds-idrepo-0 13319m 4953Mi ds-idrepo-1 2032m 2171Mi ds-idrepo-2 2033m 2152Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6491m 2245Mi 22:07:04 DEBUG --- stderr --- 22:07:04 DEBUG 22:07:04 INFO 22:07:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:07:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:07:04 INFO [loop_until]: OK (rc = 0) 22:07:04 DEBUG --- stdout --- 22:07:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5091Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11022m 36% 10681Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13081m 43% 10695Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17531m 58% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2034m 6% 4989Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16406m 54% 12737Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2959Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12481m 41% 10565Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2119m 7% 4395Mi 3% gke-xlou-cdm-ds-0f601751-j488 10719m 35% 10240Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2461Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16982m 56% 10856Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13750m 46% 7107Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6856m 43% 6116Mi 10% 22:07:04 DEBUG --- stderr --- 22:07:04 DEBUG 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215283" 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215283" 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215283" 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215283" 22:08:03 ERROR [http_cmd]: ERROR 22:08:03 ERROR --- status code --- 22:08:03 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:08:03 ERROR --- http response --- 22:08:03 INFO [http_cmd]: http status code OK 22:08:03 DEBUG --- status code --- 22:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:03 DEBUG --- http response --- 22:08:03 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:08:03 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:03 DEBUG ----- output ----- 22:08:03 DEBUG { 22:08:03 DEBUG "status": "success", 22:08:03 DEBUG "data": { 22:08:03 DEBUG "resultType": "vector", 22:08:03 DEBUG "result": [ 22:08:03 DEBUG { 22:08:03 DEBUG "metric": { 22:08:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG [...] 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 1713215283, 22:08:03 DEBUG "2286.5666666666666" 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG } 22:08:03 INFO [http_cmd]: http status code OK 22:08:03 DEBUG --- status code --- 22:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:03 DEBUG --- http response --- 22:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:03 DEBUG ----- output ----- 22:08:03 DEBUG { 22:08:03 DEBUG "status": "success", 22:08:03 DEBUG "data": { 22:08:03 DEBUG "resultType": "vector", 22:08:03 DEBUG "result": [ 22:08:03 DEBUG { 22:08:03 DEBUG "metric": { 22:08:03 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG [...] 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 1713215283, 22:08:03 DEBUG "2308.3" 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG } 22:08:03 INFO [http_cmd]: http status code OK 22:08:03 DEBUG --- status code --- 22:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:03 DEBUG --- http response --- 22:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:03 DEBUG ----- output ----- 22:08:03 DEBUG { 22:08:03 DEBUG "status": "success", 22:08:03 DEBUG "data": { 22:08:03 DEBUG "resultType": "vector", 22:08:03 DEBUG "result": [ 22:08:03 DEBUG { 22:08:03 DEBUG "metric": { 22:08:03 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG [...] 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 1713215283, 22:08:03 DEBUG "2453016576" 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG } 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215283" 22:08:03 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215283" 22:08:03 INFO [http_cmd]: http status code OK 22:08:03 DEBUG --- status code --- 22:08:03 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:03 DEBUG --- http response --- 22:08:03 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:03 DEBUG ----- output ----- 22:08:03 DEBUG { 22:08:03 DEBUG "status": "success", 22:08:03 DEBUG "data": { 22:08:03 DEBUG "resultType": "vector", 22:08:03 DEBUG "result": [ 22:08:03 DEBUG { 22:08:03 DEBUG "metric": { 22:08:03 DEBUG "instance": "10.142.0.107:9100" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG [...] 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG 22:08:03 DEBUG "instance": "10.142.0.43:9100" 22:08:03 DEBUG }, 22:08:03 DEBUG "value": [ 22:08:03 DEBUG 1713215283, 22:08:03 DEBUG "11501.23333333333" 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG ] 22:08:03 DEBUG } 22:08:03 DEBUG } 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "instance": "10.142.0.107:9100" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "instance": "10.142.0.43:9100" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215283, 22:08:04 DEBUG "3236.333333333333" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "0.078" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "ds-idrepo-2" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "ds-idrepo-0" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "0" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "ds-idrepo-2" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "ds-idrepo-0" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "0" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "ds-idrepo-2" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "ds-cts-2" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "0" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "0" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "2307.8333333333335" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "412478.5" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215284" 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG "result": [ 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG { 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG "metric": { 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG "pod": "ds-idrepo-2" 22:08:04 DEBUG }, 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:04 DEBUG "value": [ 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG 22:08:04 DEBUG { 22:08:04 DEBUG 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG 22:08:04 DEBUG "data": { 22:08:04 DEBUG [...] 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG 22:08:04 DEBUG "result": [ 22:08:04 DEBUG 22:08:04 DEBUG { 22:08:04 DEBUG 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "ds-idrepo-0" 22:08:04 DEBUG "instance": "10.142.0.107:9100" 22:08:04 DEBUG }, 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG 22:08:04 DEBUG "0" 22:08:04 DEBUG 22:08:04 DEBUG ] 22:08:04 DEBUG 22:08:04 DEBUG } 22:08:04 DEBUG [...] 22:08:04 DEBUG ] 22:08:04 DEBUG 22:08:04 DEBUG } 22:08:04 DEBUG 22:08:04 DEBUG } 22:08:04 DEBUG 22:08:04 DEBUG "instance": "10.142.0.43:9100" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "0.00040000000000001514" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG --- http response --- 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "instance": "10.142.0.107:9100" 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 DEBUG 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215284" 22:08:04 DEBUG 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG "instance": "10.142.0.43:9100" 22:08:04 DEBUG }, 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG "value": [ 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG --- http response --- 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG 1713215284, 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG "0" 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG ] 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG { 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG { 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG { 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG } 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG ] 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:08:04 DEBUG "data": { 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:04 DEBUG "data": { 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:08:04 INFO [http_cmd]: http status code OK 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:08:04 DEBUG "data": { 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:08:04 DEBUG } 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG --- status code --- 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG } 22:08:04 DEBUG { 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG { 22:08:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:08:04 DEBUG { 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG --- http response --- 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "data": { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "data": { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "data": { 22:08:04 DEBUG "data": { 22:08:04 DEBUG "data": { 22:08:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "data": { 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "instance": "10.142.0.107:9100" 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG ----- output ----- 22:08:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG "result": [ 22:08:04 DEBUG }, 22:08:04 DEBUG "result": [ 22:08:04 DEBUG }, 22:08:04 DEBUG "result": [ 22:08:04 DEBUG "result": [ 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG }, 22:08:04 DEBUG "result": [ 22:08:04 DEBUG { 22:08:04 DEBUG "value": [ 22:08:04 DEBUG { 22:08:04 DEBUG "value": [ 22:08:04 DEBUG { 22:08:04 DEBUG { 22:08:04 DEBUG { 22:08:04 DEBUG "status": "success", 22:08:04 DEBUG "value": [ 22:08:04 DEBUG { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG 22:08:04 DEBUG "metric": { 22:08:04 DEBUG 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "data": { 22:08:04 DEBUG 22:08:04 DEBUG "metric": { 22:08:04 DEBUG "pod": "ds-idrepo-2" 22:08:04 DEBUG 22:08:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:08:04 DEBUG 22:08:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:08:04 DEBUG "pod": "ds-idrepo-2" 22:08:04 DEBUG "pod": "ds-idrepo-2" 22:08:04 DEBUG "resultType": "vector", 22:08:04 DEBUG 22:08:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:08:04 DEBUG }, 22:08:04 DEBUG 22:08:04 DEBUG }, 22:08:04 DEBUG 22:08:04 DEBUG }, 22:08:04 DEBUG }, 22:08:04 DEBUG }, 22:08:04 DEBUG "result": [ 22:08:04 DEBUG 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG [...] 22:08:04 DEBUG "value": [ 22:08:04 DEBUG [...] 22:08:04 DEBUG "value": [ 22:08:04 DEBUG "value": [ 22:08:04 DEBUG "value": [ 22:08:04 DEBUG { 22:08:04 DEBUG [...] 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "metric": { 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG }, 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG [...] 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG [...] 22:08:04 DEBUG "instance": "10.142.0.43:9100" 22:08:04 DEBUG [...] 22:08:04 DEBUG [...] 22:08:04 DEBUG [...] 22:08:04 DEBUG "value": [ 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG [...] 22:08:04 DEBUG 22:08:04 DEBUG }, 22:08:04 DEBUG 22:08:04 DEBUG }, 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG }, 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 INFO 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 DEBUG 22:08:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG 22:08:04 DEBUG "pod": "ds-idrepo-0" 22:08:04 DEBUG "2304.1666666666665" 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG "13858241536" 22:08:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:08:04 DEBUG "pod": "ds-idrepo-0" 22:08:04 DEBUG "pod": "ds-idrepo-0" 22:08:04 DEBUG [...] 22:08:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:08:04 DEBUG "15.935371812129693" 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG }, 22:08:04 DEBUG ] 22:08:04 DEBUG }, 22:08:04 DEBUG ] 22:08:04 DEBUG }, 22:08:04 DEBUG }, 22:08:04 DEBUG }, 22:08:04 DEBUG 22:08:04 DEBUG ] 22:08:04 DEBUG }, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG } 22:08:04 DEBUG "value": [ 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG "value": [ 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 22:08:04 DEBUG } 22:08:04 DEBUG "value": [ 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG ] 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "value": [ 22:08:04 DEBUG } 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG 22:08:04 DEBUG ] 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG "0" 22:08:04 DEBUG } 22:08:04 DEBUG "267036.48997708014" 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG } 22:08:04 DEBUG "6.74662945496039" 22:08:04 DEBUG "0" 22:08:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:08:04 DEBUG }, 22:08:04 DEBUG "9903000.54571252" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG "0" 22:08:04 DEBUG ] 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG "value": [ 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG 1713215284, 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG ] 22:08:04 DEBUG "0" 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 DEBUG ] 22:08:04 DEBUG } 22:08:04 DEBUG } 22:08:04 INFO [loop_until]: OK (rc = 0) 22:08:04 DEBUG --- stdout --- 22:08:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12962m 8612Mi am-7d597cf7d6-8vr8f 11522m 9066Mi am-7d597cf7d6-9wz9h 15459m 9215Mi ds-cts-0 15648m 8437Mi ds-cts-1 10957m 8474Mi ds-cts-2 10569m 8658Mi ds-idrepo-0 13877m 4953Mi ds-idrepo-1 1986m 2233Mi ds-idrepo-2 1965m 2197Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6728m 2249Mi 22:08:04 DEBUG --- stderr --- 22:08:04 DEBUG 22:08:04 INFO 22:08:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:08:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:08:04 INFO [loop_until]: OK (rc = 0) 22:08:04 DEBUG --- stdout --- 22:08:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5085Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10844m 36% 10692Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13381m 44% 10690Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17624m 59% 3073Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2043m 6% 5042Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16637m 55% 12748Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2960Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12548m 42% 10568Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2133m 7% 4461Mi 3% gke-xlou-cdm-ds-0f601751-j488 10716m 35% 10232Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2461Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17027m 57% 10900Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13841m 46% 7124Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6903m 43% 6158Mi 10% 22:08:04 DEBUG --- stderr --- 22:08:04 DEBUG 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215344" 22:09:04 ERROR [http_cmd]: ERROR 22:09:04 ERROR --- status code --- 22:09:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:09:04 ERROR --- http response --- 22:09:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:09:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "2296.6" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG { 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "data": { 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG "result": [ 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG { 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:09:04 DEBUG "instance": "10.142.0.107:9100" 22:09:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:09:04 DEBUG }, 22:09:04 DEBUG }, 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG [...] 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:09:04 DEBUG "instance": "10.142.0.43:9100" 22:09:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:09:04 DEBUG }, 22:09:04 DEBUG }, 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG 1713215344, 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG "2498510848" 22:09:04 DEBUG "11441.099999999997" 22:09:04 DEBUG "2278" 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG ] 22:09:04 DEBUG ] 22:09:04 DEBUG ] 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG ] 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG ] 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG { 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG "data": { 22:09:04 DEBUG { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG "data": { 22:09:04 DEBUG { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "result": [ 22:09:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:09:04 DEBUG { 22:09:04 DEBUG }, 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "instance": "10.142.0.107:9100" 22:09:04 DEBUG 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:09:04 DEBUG 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "instance": "10.142.0.43:9100" 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG }, 22:09:04 DEBUG "0.078" 22:09:04 DEBUG "value": [ 22:09:04 DEBUG ] 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG } 22:09:04 DEBUG "3109.8666666666663" 22:09:04 DEBUG ] 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG { 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "ds-idrepo-2" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "ds-idrepo-0" 22:09:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "ds-idrepo-2" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "ds-cts-2" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "ds-idrepo-2" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "ds-idrepo-0" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "2295.4333333333334" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "412948" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215344" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "ds-idrepo-2" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG }, 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG "value": [ 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "ds-idrepo-0" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "instance": "10.142.0.107:9100" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "instance": "10.142.0.43:9100" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0.00030000000000001136" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215344" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "ds-idrepo-2" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "ds-idrepo-0" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "instance": "10.142.0.107:9100" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "instance": "10.142.0.43:9100" 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "0" 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215344" 22:09:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215344" 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG --- http response --- 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG --- http response --- 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG { 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG { 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG "data": { 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG "data": { 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG { 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG { 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "data": { 22:09:04 DEBUG { 22:09:04 DEBUG "data": { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:09:04 DEBUG "data": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:09:04 DEBUG ----- output ----- 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "result": [ 22:09:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:09:04 DEBUG "result": [ 22:09:04 DEBUG }, 22:09:04 DEBUG { 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG "result": [ 22:09:04 DEBUG { 22:09:04 DEBUG }, 22:09:04 DEBUG { 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG { 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "value": [ 22:09:04 INFO [http_cmd]: http status code OK 22:09:04 DEBUG "metric": { 22:09:04 DEBUG 22:09:04 DEBUG "data": { 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG "metric": { 22:09:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:09:04 DEBUG 22:09:04 DEBUG --- status code --- 22:09:04 DEBUG "instance": "10.142.0.107:9100" 22:09:04 DEBUG 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "pod": "ds-idrepo-2" 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG "result": [ 22:09:04 DEBUG }, 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG --- http response --- 22:09:04 DEBUG "value": [ 22:09:04 DEBUG [...] 22:09:04 DEBUG { 22:09:04 DEBUG "value": [ 22:09:04 DEBUG { 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:09:04 DEBUG 22:09:04 DEBUG "metric": { 22:09:04 DEBUG 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG ----- output ----- 22:09:04 DEBUG 22:09:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:09:04 DEBUG 22:09:04 DEBUG "data": { 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG { 22:09:04 DEBUG 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG [...] 22:09:04 DEBUG 22:09:04 DEBUG [...] 22:09:04 DEBUG "status": "success", 22:09:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:09:04 DEBUG "value": [ 22:09:04 DEBUG [...] 22:09:04 DEBUG "result": [ 22:09:04 DEBUG 22:09:04 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:09:04 DEBUG 22:09:04 DEBUG "data": { 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG { 22:09:04 DEBUG 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG "resultType": "vector", 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "metric": { 22:09:04 DEBUG 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG "result": [ 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG 22:09:04 DEBUG 22:09:04 DEBUG "pod": "ds-idrepo-2" 22:09:04 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG "instance": "10.142.0.43:9100" 22:09:04 DEBUG { 22:09:04 DEBUG "2295.1666666666665" 22:09:04 DEBUG [...] 22:09:04 DEBUG "pod": "ds-idrepo-0" 22:09:04 DEBUG }, 22:09:04 DEBUG }, 22:09:04 DEBUG "0.0010142309967926242" 22:09:04 DEBUG }, 22:09:04 DEBUG "metric": { 22:09:04 DEBUG ] 22:09:04 DEBUG 22:09:04 DEBUG }, 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "value": [ 22:09:04 DEBUG ] 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:09:04 DEBUG } 22:09:04 DEBUG 22:09:04 DEBUG "value": [ 22:09:04 DEBUG 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG } 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG }, 22:09:04 DEBUG ] 22:09:04 DEBUG 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG 22:09:04 DEBUG "2066430.9158284806" 22:09:04 DEBUG ] 22:09:04 DEBUG "13858896896" 22:09:04 DEBUG "value": [ 22:09:04 DEBUG } 22:09:04 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:09:04 DEBUG "0" 22:09:04 DEBUG 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG 22:09:04 DEBUG } 22:09:04 DEBUG }, 22:09:04 DEBUG ] 22:09:04 DEBUG [...] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG 22:09:04 DEBUG "value": [ 22:09:04 DEBUG } 22:09:04 DEBUG 22:09:04 DEBUG ] 22:09:04 DEBUG ] 22:09:04 DEBUG 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG ] 22:09:04 DEBUG 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG [...] 22:09:04 DEBUG "4081.5868962725376" 22:09:04 DEBUG } 22:09:04 DEBUG 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG "pod": "ds-idrepo-0" 22:09:04 DEBUG 22:09:04 DEBUG } 22:09:04 DEBUG }, 22:09:04 DEBUG 22:09:04 DEBUG ] 22:09:04 DEBUG "value": [ 22:09:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:09:04 DEBUG } 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG }, 22:09:04 DEBUG } 22:09:04 DEBUG "0" 22:09:04 DEBUG "value": [ 22:09:04 DEBUG ] 22:09:04 DEBUG 1713215344, 22:09:04 DEBUG } 22:09:04 DEBUG "6.735436379398402" 22:09:04 DEBUG ] 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 DEBUG ] 22:09:04 DEBUG } 22:09:04 DEBUG } 22:09:04 INFO 22:09:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:09:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:09:04 INFO [loop_until]: OK (rc = 0) 22:09:04 DEBUG --- stdout --- 22:09:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12874m 8612Mi am-7d597cf7d6-8vr8f 11548m 9065Mi am-7d597cf7d6-9wz9h 16360m 9266Mi ds-cts-0 14816m 8429Mi ds-cts-1 10958m 8477Mi ds-cts-2 10679m 8662Mi ds-idrepo-0 13747m 4953Mi ds-idrepo-1 2054m 2246Mi ds-idrepo-2 2003m 2243Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 1m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6697m 2290Mi 22:09:04 DEBUG --- stderr --- 22:09:04 DEBUG 22:09:04 INFO 22:09:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:09:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:09:04 INFO [loop_until]: OK (rc = 0) 22:09:04 DEBUG --- stdout --- 22:09:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5063Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10941m 36% 10679Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13082m 43% 10685Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17543m 58% 3056Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2060m 6% 5090Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16317m 54% 12738Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2955Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12551m 42% 10560Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2088m 6% 4493Mi 4% gke-xlou-cdm-ds-0f601751-j488 10712m 35% 10242Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2463Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17109m 57% 10889Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13768m 46% 7114Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6895m 43% 6156Mi 10% 22:09:04 DEBUG --- stderr --- 22:09:04 DEBUG 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 ERROR [http_cmd]: ERROR 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 ERROR --- status code --- 22:10:04 DEBUG --- status code --- 22:10:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 ERROR --- http response --- 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:10:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "2286.633333333333" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "2281.1666666666665" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0.078" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "2500825088" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG { 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "instance": "10.142.0.107:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "instance": "10.142.0.43:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "11094.166666666664" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "instance": "10.142.0.107:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "instance": "10.142.0.43:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "3102.5333333333333" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "ds-idrepo-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "ds-cts-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "ds-idrepo-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "ds-idrepo-0" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "ds-idrepo-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "ds-idrepo-0" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG 22:10:04 DEBUG { 22:10:04 DEBUG 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG "data": { 22:10:04 DEBUG }, 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "value": [ 22:10:04 DEBUG "result": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG { 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG "2285.9" 22:10:04 DEBUG "metric": { 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG ] 22:10:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG } 22:10:04 DEBUG }, 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG ] 22:10:04 DEBUG "value": [ 22:10:04 DEBUG } 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG 22:10:04 DEBUG } 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG 22:10:04 DEBUG { 22:10:04 DEBUG 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG [...] 22:10:04 DEBUG "data": { 22:10:04 DEBUG 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG 22:10:04 DEBUG "result": [ 22:10:04 DEBUG 22:10:04 DEBUG { 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG "metric": { 22:10:04 DEBUG }, 22:10:04 DEBUG "instance": "10.142.0.107:9100" 22:10:04 DEBUG "value": [ 22:10:04 DEBUG }, 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG "0" 22:10:04 DEBUG 22:10:04 DEBUG ] 22:10:04 DEBUG 22:10:04 DEBUG } 22:10:04 DEBUG 22:10:04 DEBUG ] 22:10:04 DEBUG [...] 22:10:04 DEBUG } 22:10:04 DEBUG 22:10:04 DEBUG } 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "instance": "10.142.0.43:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0.0002666666666669926" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "ds-idrepo-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "ds-idrepo-0" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "413243.5" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "8735583.88861613" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215404" 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "ds-idrepo-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "ds-idrepo-0" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "instance": "10.142.0.107:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "instance": "10.142.0.43:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "ds-idrepo-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "ds-idrepo-0" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215404" 22:10:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215404" 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "2281.733333333333" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "136176.49223854713" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "instance": "10.142.0.107:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "instance": "10.142.0.43:9100" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "13859478528" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG { 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "11.27320227956976" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "pod": "ds-idrepo-2" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "pod": "ds-idrepo-0" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "0" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO [http_cmd]: http status code OK 22:10:04 DEBUG --- status code --- 22:10:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:10:04 DEBUG --- http response --- 22:10:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:10:04 DEBUG ----- output ----- 22:10:04 DEBUG { 22:10:04 DEBUG "status": "success", 22:10:04 DEBUG "data": { 22:10:04 DEBUG "resultType": "vector", 22:10:04 DEBUG "result": [ 22:10:04 DEBUG { 22:10:04 DEBUG "metric": { 22:10:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG [...] 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG 22:10:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:10:04 DEBUG }, 22:10:04 DEBUG "value": [ 22:10:04 DEBUG 1713215404, 22:10:04 DEBUG "6.722822023802388" 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG ] 22:10:04 DEBUG } 22:10:04 DEBUG } 22:10:04 INFO 22:10:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:10:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:10:04 INFO [loop_until]: OK (rc = 0) 22:10:04 DEBUG --- stdout --- 22:10:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12535m 8614Mi am-7d597cf7d6-8vr8f 12063m 9066Mi am-7d597cf7d6-9wz9h 15991m 9231Mi ds-cts-0 15872m 8427Mi ds-cts-1 11317m 8505Mi ds-cts-2 10675m 8666Mi ds-idrepo-0 13456m 4955Mi ds-idrepo-1 2010m 2287Mi ds-idrepo-2 1977m 2280Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6906m 2292Mi 22:10:04 DEBUG --- stderr --- 22:10:04 DEBUG 22:10:04 INFO 22:10:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:10:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:10:04 INFO [loop_until]: OK (rc = 0) 22:10:04 DEBUG --- stdout --- 22:10:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11010m 36% 10691Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13196m 44% 10696Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17526m 58% 3048Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2095m 7% 5112Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16362m 54% 12737Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2956Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12220m 40% 10564Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2096m 7% 4531Mi 4% gke-xlou-cdm-ds-0f601751-j488 10614m 35% 10251Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17076m 57% 10881Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13791m 46% 7126Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6840m 43% 6157Mi 10% 22:10:04 DEBUG --- stderr --- 22:10:04 DEBUG 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "2302.866666666667" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215464" 22:11:04 ERROR [http_cmd]: ERROR 22:11:04 ERROR --- status code --- 22:11:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:11:04 ERROR --- http response --- 22:11:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:11:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0.078" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "2309.1" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "2503000064" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "instance": "10.142.0.107:9100" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "instance": "10.142.0.43:9100" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "3091.733333333333" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "instance": "10.142.0.107:9100" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "instance": "10.142.0.43:9100" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "11138.966666666665" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "ds-idrepo-2" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "ds-cts-2" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "ds-idrepo-2" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "ds-idrepo-0" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "ds-idrepo-2" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "ds-idrepo-0" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG { 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "instance": "10.142.0.107:9100" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG 22:11:04 DEBUG { 22:11:04 DEBUG [...] 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG 22:11:04 DEBUG "data": { 22:11:04 DEBUG 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG 22:11:04 DEBUG "result": [ 22:11:04 DEBUG "instance": "10.142.0.43:9100" 22:11:04 DEBUG { 22:11:04 DEBUG }, 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "value": [ 22:11:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG }, 22:11:04 DEBUG "0.00043333333333350765" 22:11:04 DEBUG "value": [ 22:11:04 DEBUG ] 22:11:04 DEBUG 22:11:04 DEBUG } 22:11:04 DEBUG 22:11:04 DEBUG ] 22:11:04 DEBUG 22:11:04 DEBUG } 22:11:04 DEBUG [...] 22:11:04 DEBUG } 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "2302.8" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:11:04 DEBUG "pod": "ds-idrepo-2" 22:11:04 DEBUG }, 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:11:04 DEBUG "pod": "ds-idrepo-0" 22:11:04 DEBUG }, 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "412856" 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "instance": "10.142.0.107:9100" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "instance": "10.142.0.43:9100" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "494.6859903381643" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "2301.366666666667" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "ds-idrepo-2" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "ds-idrepo-0" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215464" 22:11:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215464" 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "ds-idrepo-2" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "ds-idrepo-0" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "0" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "32653480.193236716" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG { 22:11:04 DEBUG { 22:11:04 DEBUG { 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG "data": { 22:11:04 DEBUG "data": { 22:11:04 DEBUG "data": { 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG "result": [ 22:11:04 DEBUG "result": [ 22:11:04 DEBUG "result": [ 22:11:04 DEBUG { 22:11:04 DEBUG { 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "instance": "10.142.0.107:9100" 22:11:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:11:04 DEBUG "pod": "ds-idrepo-2" 22:11:04 DEBUG }, 22:11:04 DEBUG }, 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG "value": [ 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG [...] 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 INFO [http_cmd]: http status code OK 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG --- status code --- 22:11:04 DEBUG 22:11:04 DEBUG "instance": "10.142.0.43:9100" 22:11:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:11:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:11:04 DEBUG "pod": "ds-idrepo-0" 22:11:04 DEBUG }, 22:11:04 DEBUG }, 22:11:04 DEBUG --- http response --- 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG "value": [ 22:11:04 DEBUG "value": [ 22:11:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG ----- output ----- 22:11:04 DEBUG "13860049920" 22:11:04 DEBUG "0.0035808574879233083" 22:11:04 DEBUG "0" 22:11:04 DEBUG { 22:11:04 DEBUG ] 22:11:04 DEBUG ] 22:11:04 DEBUG ] 22:11:04 DEBUG "status": "success", 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG "data": { 22:11:04 DEBUG ] 22:11:04 DEBUG ] 22:11:04 DEBUG ] 22:11:04 DEBUG "resultType": "vector", 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG "result": [ 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 DEBUG { 22:11:04 DEBUG "metric": { 22:11:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG [...] 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG 22:11:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:11:04 DEBUG }, 22:11:04 DEBUG "value": [ 22:11:04 DEBUG 1713215464, 22:11:04 DEBUG "6.669645918977977" 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG ] 22:11:04 DEBUG } 22:11:04 DEBUG } 22:11:04 INFO 22:11:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:11:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:11:04 INFO [loop_until]: OK (rc = 0) 22:11:04 DEBUG --- stdout --- 22:11:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13318m 8612Mi am-7d597cf7d6-8vr8f 11900m 9063Mi am-7d597cf7d6-9wz9h 15743m 9232Mi ds-cts-0 15120m 8432Mi ds-cts-1 11348m 8491Mi ds-cts-2 10874m 8667Mi ds-idrepo-0 13850m 4955Mi ds-idrepo-1 2031m 2317Mi ds-idrepo-2 1967m 2309Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6637m 2294Mi 22:11:04 DEBUG --- stderr --- 22:11:04 DEBUG 22:11:04 INFO 22:11:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:11:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:11:04 INFO [loop_until]: OK (rc = 0) 22:11:04 DEBUG --- stdout --- 22:11:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5074Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10686m 35% 10698Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13182m 44% 10694Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17680m 59% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2070m 6% 5151Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16544m 55% 12763Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2951Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12565m 42% 10556Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2149m 7% 4560Mi 4% gke-xlou-cdm-ds-0f601751-j488 10727m 35% 10246Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17193m 57% 10879Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13866m 46% 7119Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6863m 43% 6166Mi 10% 22:11:04 DEBUG --- stderr --- 22:11:04 DEBUG 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "2288.6666666666665" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0.079" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215524" 22:12:04 ERROR [http_cmd]: ERROR 22:12:04 ERROR --- status code --- 22:12:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:12:04 ERROR --- http response --- 22:12:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:12:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "instance": "10.142.0.107:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "instance": "10.142.0.43:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "11053.199999999997" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG --- http response --- 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "2294.8" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "ds-idrepo-2" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "ds-cts-2" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "2503839744" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "instance": "10.142.0.107:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "instance": "10.142.0.43:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "3240.233333333333" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "ds-idrepo-2" 22:12:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:12:04 DEBUG }, 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "ds-idrepo-0" 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0" 22:12:04 DEBUG "0" 22:12:04 DEBUG ] 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "ds-idrepo-2" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "ds-idrepo-0" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "ds-idrepo-2" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "ds-idrepo-0" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "2289.366666666667" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "instance": "10.142.0.107:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "instance": "10.142.0.43:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0.00043333333333350765" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "412805" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "instance": "10.142.0.107:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "instance": "10.142.0.43:9100" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0.00033333333333333365" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "4884058.478361133" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG { 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "ds-idrepo-2" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "ds-idrepo-0" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "2304.9333333333334" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "ds-idrepo-2" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "ds-idrepo-0" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "0" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215524" 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "4174.957931266092" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "6.693989771134241" 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 INFO [http_cmd]: http status code OK 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- status code --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG --- http response --- 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:12:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG ----- output ----- 22:12:04 DEBUG { 22:12:04 DEBUG { 22:12:04 DEBUG { 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "status": "success", 22:12:04 DEBUG "data": { 22:12:04 DEBUG "data": { 22:12:04 DEBUG "data": { 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "resultType": "vector", 22:12:04 DEBUG "result": [ 22:12:04 DEBUG "result": [ 22:12:04 DEBUG "result": [ 22:12:04 DEBUG { 22:12:04 DEBUG { 22:12:04 DEBUG { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "metric": { 22:12:04 DEBUG "instance": "10.142.0.107:9100" 22:12:04 DEBUG "pod": "ds-idrepo-2" 22:12:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:12:04 DEBUG }, 22:12:04 DEBUG }, 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG "value": [ 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG [...] 22:12:04 DEBUG [...] 22:12:04 DEBUG [...] 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG 22:12:04 DEBUG "instance": "10.142.0.43:9100" 22:12:04 DEBUG "pod": "ds-idrepo-0" 22:12:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:12:04 DEBUG }, 22:12:04 DEBUG }, 22:12:04 DEBUG }, 22:12:04 DEBUG "value": [ 22:12:04 DEBUG "value": [ 22:12:04 DEBUG "value": [ 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG 1713215524, 22:12:04 DEBUG "13860699136" 22:12:04 DEBUG "0" 22:12:04 DEBUG "7.5315816642707905" 22:12:04 DEBUG ] 22:12:04 DEBUG ] 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG ] 22:12:04 DEBUG ] 22:12:04 DEBUG ] 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 DEBUG } 22:12:04 INFO 22:12:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:12:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:12:04 INFO [loop_until]: OK (rc = 0) 22:12:04 DEBUG --- stdout --- 22:12:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12655m 8616Mi am-7d597cf7d6-8vr8f 11968m 9066Mi am-7d597cf7d6-9wz9h 16621m 9269Mi ds-cts-0 15182m 8451Mi ds-cts-1 11074m 8484Mi ds-cts-2 10623m 8696Mi ds-idrepo-0 13274m 4955Mi ds-idrepo-1 1972m 2344Mi ds-idrepo-2 1963m 2335Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6733m 2348Mi 22:12:04 DEBUG --- stderr --- 22:12:04 DEBUG 22:12:04 INFO 22:12:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:12:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:12:04 INFO [loop_until]: OK (rc = 0) 22:12:04 DEBUG --- stdout --- 22:12:04 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5065Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10919m 36% 10698Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13059m 43% 10690Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17633m 59% 3064Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2041m 6% 5182Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16137m 54% 12760Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2951Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12525m 41% 10555Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2100m 7% 4574Mi 4% gke-xlou-cdm-ds-0f601751-j488 10712m 35% 10273Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2448Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17280m 57% 10914Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13368m 44% 7118Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6665m 41% 6209Mi 10% 22:12:04 DEBUG --- stderr --- 22:12:04 DEBUG 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "2286.9" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0.079" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215584" 22:13:04 ERROR [http_cmd]: ERROR 22:13:04 ERROR --- status code --- 22:13:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:13:04 ERROR --- http response --- 22:13:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:13:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "instance": "10.142.0.107:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "instance": "10.142.0.43:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "11020.833333333332" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-idrepo-2" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "ds-cts-2" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "instance": "10.142.0.107:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "instance": "10.142.0.43:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "3046.766666666667" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "2284.5" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "2561253376" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-idrepo-2" 22:13:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:13:04 DEBUG }, 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "ds-idrepo-0" 22:13:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:13:04 DEBUG }, 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-idrepo-2" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "ds-idrepo-0" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "413254.5" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-idrepo-2" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "ds-idrepo-0" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "instance": "10.142.0.107:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "instance": "10.142.0.43:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0.00036666666666652263" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "2286.9" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-cts-1" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "instance": "10.142.0.107:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "instance": "10.142.0.43:9100" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-cts-1" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "2287.1" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-idrepo-2" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "ds-idrepo-0" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "6.686267309708571" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-cts-1" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG { 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "data": { 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG "result": [ 22:13:04 DEBUG { 22:13:04 DEBUG "metric": { 22:13:04 DEBUG "pod": "ds-idrepo-2" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "pod": "ds-idrepo-0" 22:13:04 DEBUG }, 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG "0" 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215584" 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG --- http response --- 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG ----- output ----- 22:13:04 INFO [http_cmd]: http status code OK 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG { 22:13:04 DEBUG --- status code --- 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:13:04 DEBUG "data": { 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:13:04 DEBUG --- http response --- 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG "result": [ 22:13:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:13:04 DEBUG { 22:13:04 DEBUG { 22:13:04 DEBUG ----- output ----- 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "metric": { 22:13:04 DEBUG { 22:13:04 DEBUG "data": { 22:13:04 DEBUG "instance": "10.142.0.107:9100" 22:13:04 DEBUG "status": "success", 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG }, 22:13:04 DEBUG "data": { 22:13:04 DEBUG "result": [ 22:13:04 DEBUG "value": [ 22:13:04 DEBUG "resultType": "vector", 22:13:04 DEBUG { 22:13:04 DEBUG 22:13:04 DEBUG "result": [ 22:13:04 DEBUG "metric": { 22:13:04 DEBUG 22:13:04 DEBUG { 22:13:04 DEBUG "pod": "ds-idrepo-2" 22:13:04 DEBUG 22:13:04 DEBUG "metric": { 22:13:04 DEBUG }, 22:13:04 DEBUG [...] 22:13:04 DEBUG "pod": "ds-cts-1" 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG }, 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "value": [ 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "instance": "10.142.0.43:9100" 22:13:04 DEBUG 22:13:04 DEBUG [...] 22:13:04 DEBUG }, 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "value": [ 22:13:04 DEBUG [...] 22:13:04 DEBUG 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG 22:13:04 DEBUG 22:13:04 DEBUG "13861288960" 22:13:04 DEBUG 22:13:04 DEBUG "pod": "ds-idrepo-0" 22:13:04 DEBUG ] 22:13:04 DEBUG 22:13:04 DEBUG }, 22:13:04 DEBUG } 22:13:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:13:04 DEBUG "value": [ 22:13:04 DEBUG ] 22:13:04 DEBUG }, 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG } 22:13:04 DEBUG "value": [ 22:13:04 DEBUG "0" 22:13:04 DEBUG } 22:13:04 DEBUG 1713215584, 22:13:04 DEBUG ] 22:13:04 DEBUG "0.000020548529202779797" 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 DEBUG ] 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 DEBUG } 22:13:04 INFO 22:13:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:13:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:13:04 INFO [loop_until]: OK (rc = 0) 22:13:04 DEBUG --- stdout --- 22:13:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12836m 8613Mi am-7d597cf7d6-8vr8f 12040m 9066Mi am-7d597cf7d6-9wz9h 15647m 9250Mi ds-cts-0 15440m 8448Mi ds-cts-1 11118m 8479Mi ds-cts-2 10775m 8683Mi ds-idrepo-0 13454m 4955Mi ds-idrepo-1 1981m 2381Mi ds-idrepo-2 1993m 2371Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6612m 2348Mi 22:13:04 DEBUG --- stderr --- 22:13:04 DEBUG 22:13:04 INFO 22:13:04 INFO [loop_until]: kubectl --namespace=xlou top node 22:13:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:13:05 INFO [loop_until]: OK (rc = 0) 22:13:05 DEBUG --- stdout --- 22:13:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10971m 36% 10705Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13104m 43% 10692Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17480m 58% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2044m 6% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16172m 54% 12745Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2943Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12583m 42% 10571Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2085m 6% 4622Mi 4% gke-xlou-cdm-ds-0f601751-j488 10223m 34% 10260Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 57m 0% 2455Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17137m 57% 10895Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13741m 46% 7126Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6826m 42% 6212Mi 10% 22:13:05 DEBUG --- stderr --- 22:13:05 DEBUG 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "instance": "10.142.0.107:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "instance": "10.142.0.43:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "11553.033333333331" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "2302.2" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "ds-idrepo-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-cts-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0.079" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215644" 22:14:04 ERROR [http_cmd]: ERROR 22:14:04 ERROR --- status code --- 22:14:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:14:04 ERROR --- http response --- 22:14:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:14:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "2562162688" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "instance": "10.142.0.107:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "instance": "10.142.0.43:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "4586.133333333333" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "2302.9666666666667" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "ds-idrepo-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-idrepo-0" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "413526.5" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "ds-idrepo-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-idrepo-0" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "ds-idrepo-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-idrepo-0" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG "instance": "10.142.0.107:9100" 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG }, 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG "value": [ 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:14:04 DEBUG [...] 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG 22:14:04 DEBUG { 22:14:04 DEBUG 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG 22:14:04 DEBUG "data": { 22:14:04 DEBUG "instance": "10.142.0.43:9100" 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG }, 22:14:04 DEBUG "result": [ 22:14:04 DEBUG "value": [ 22:14:04 DEBUG { 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "0.0003333333333330302" 22:14:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:14:04 DEBUG ] 22:14:04 DEBUG }, 22:14:04 DEBUG } 22:14:04 DEBUG "value": [ 22:14:04 DEBUG ] 22:14:04 DEBUG 22:14:04 DEBUG } 22:14:04 DEBUG 22:14:04 DEBUG } 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-cts-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "51537360.28861206" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "2301.8" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "instance": "10.142.0.107:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "instance": "10.142.0.43:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-cts-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "34036725.61432169" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "ds-idrepo-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-idrepo-0" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "2291.8333333333335" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "ds-idrepo-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-idrepo-0" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "6.730260619983219" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "ds-idrepo-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-idrepo-0" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "0" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-cts-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "309655759.85297114" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "pod": "ds-cts-2" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "10.75408015111296" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215644" 22:14:04 INFO [http_cmd]: http status code OK 22:14:04 DEBUG --- status code --- 22:14:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:14:04 DEBUG --- http response --- 22:14:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:14:04 DEBUG ----- output ----- 22:14:04 DEBUG { 22:14:04 DEBUG "status": "success", 22:14:04 DEBUG "data": { 22:14:04 DEBUG "resultType": "vector", 22:14:04 DEBUG "result": [ 22:14:04 DEBUG { 22:14:04 DEBUG "metric": { 22:14:04 DEBUG "instance": "10.142.0.107:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG [...] 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG 22:14:04 DEBUG "instance": "10.142.0.43:9100" 22:14:04 DEBUG }, 22:14:04 DEBUG "value": [ 22:14:04 DEBUG 1713215644, 22:14:04 DEBUG "13861938176" 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG ] 22:14:04 DEBUG } 22:14:04 DEBUG } 22:14:04 INFO 22:14:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:14:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:14:04 INFO [loop_until]: OK (rc = 0) 22:14:04 DEBUG --- stdout --- 22:14:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12941m 8636Mi am-7d597cf7d6-8vr8f 12031m 9065Mi am-7d597cf7d6-9wz9h 16021m 9275Mi ds-cts-0 15293m 8444Mi ds-cts-1 11278m 8487Mi ds-cts-2 10434m 8679Mi ds-idrepo-0 13482m 4955Mi ds-idrepo-1 1975m 2390Mi ds-idrepo-2 1940m 2382Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6524m 2350Mi 22:14:04 DEBUG --- stderr --- 22:14:04 DEBUG 22:14:05 INFO 22:14:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:14:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:14:05 INFO [loop_until]: OK (rc = 0) 22:14:05 DEBUG --- stdout --- 22:14:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5065Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11170m 37% 10715Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12706m 42% 10713Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17589m 58% 3064Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1978m 6% 5223Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16425m 55% 12759Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2948Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12691m 42% 10562Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2088m 6% 4618Mi 4% gke-xlou-cdm-ds-0f601751-j488 10807m 36% 10265Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2455Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17317m 58% 10917Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13870m 46% 7122Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6910m 43% 6212Mi 10% 22:14:05 DEBUG --- stderr --- 22:14:05 DEBUG 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG { 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG }, 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG { 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG "value": [ 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG "data": { 22:15:04 DEBUG 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG "result": [ 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG [...] 22:15:04 DEBUG { 22:15:04 DEBUG { 22:15:04 DEBUG { 22:15:04 DEBUG 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG 22:15:04 DEBUG "data": { 22:15:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:15:04 DEBUG "data": { 22:15:04 DEBUG 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG }, 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:15:04 DEBUG "result": [ 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "result": [ 22:15:04 DEBUG }, 22:15:04 DEBUG { 22:15:04 DEBUG 22:15:04 DEBUG { 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "metric": { 22:15:04 DEBUG 22:15:04 DEBUG "metric": { 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "pod": "ds-idrepo-2" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG "instance": "10.142.0.107:9100" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 DEBUG "2279.4" 22:15:04 DEBUG }, 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG [...] 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG }, 22:15:04 DEBUG ] 22:15:04 DEBUG "value": [ 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG "value": [ 22:15:04 DEBUG } 22:15:04 DEBUG 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG 22:15:04 DEBUG ] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG } 22:15:04 DEBUG 22:15:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG } 22:15:04 DEBUG [...] 22:15:04 DEBUG }, 22:15:04 DEBUG { 22:15:04 DEBUG { 22:15:04 DEBUG [...] 22:15:04 DEBUG { 22:15:04 DEBUG 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "data": { 22:15:04 DEBUG "data": { 22:15:04 DEBUG 22:15:04 DEBUG "data": { 22:15:04 DEBUG 22:15:04 DEBUG "2562965504" 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "pod": "ds-cts-2" 22:15:04 DEBUG ] 22:15:04 DEBUG "result": [ 22:15:04 DEBUG "result": [ 22:15:04 DEBUG "instance": "10.142.0.43:9100" 22:15:04 DEBUG "result": [ 22:15:04 DEBUG }, 22:15:04 DEBUG } 22:15:04 DEBUG { 22:15:04 DEBUG { 22:15:04 DEBUG }, 22:15:04 DEBUG { 22:15:04 DEBUG "value": [ 22:15:04 DEBUG ] 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "metric": { 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG } 22:15:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:15:04 DEBUG "instance": "10.142.0.107:9100" 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:15:04 DEBUG "0" 22:15:04 DEBUG } 22:15:04 DEBUG }, 22:15:04 DEBUG }, 22:15:04 DEBUG "12220.666666666664" 22:15:04 DEBUG }, 22:15:04 DEBUG ] 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "value": [ 22:15:04 DEBUG ] 22:15:04 DEBUG "value": [ 22:15:04 DEBUG } 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG } 22:15:04 DEBUG 22:15:04 DEBUG ] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG ] 22:15:04 DEBUG 22:15:04 DEBUG } 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG } 22:15:04 DEBUG 22:15:04 DEBUG } 22:15:04 DEBUG [...] 22:15:04 DEBUG [...] 22:15:04 DEBUG } 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:15:04 DEBUG "instance": "10.142.0.43:9100" 22:15:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:15:04 DEBUG }, 22:15:04 DEBUG }, 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0.079" 22:15:04 DEBUG "2951.9666666666667" 22:15:04 DEBUG "2290.9333333333334" 22:15:04 DEBUG ] 22:15:04 DEBUG ] 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG ] 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215704" 22:15:04 ERROR [http_cmd]: ERROR 22:15:04 ERROR --- status code --- 22:15:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:15:04 ERROR --- http response --- 22:15:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:15:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "ds-idrepo-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-idrepo-0" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "413860.5" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "ds-idrepo-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-idrepo-0" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "instance": "10.142.0.107:9100" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "instance": "10.142.0.43:9100" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0.00020000000000000757" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "ds-idrepo-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-idrepo-0" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-cts-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "41624494.489204" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "ds-idrepo-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-idrepo-0" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "instance": "10.142.0.107:9100" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "instance": "10.142.0.43:9100" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "2280.5333333333333" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "6.736865601882063" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "2284.1" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-cts-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "32224022.68772156" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-cts-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "292845123.81566226" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "ds-idrepo-2" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-idrepo-0" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "pod": "ds-idrepo-2" 22:15:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:15:04 DEBUG }, 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "pod": "ds-idrepo-0" 22:15:04 DEBUG "pod": "ds-cts-2" 22:15:04 DEBUG }, 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "0" 22:15:04 DEBUG "10.640934482758723" 22:15:04 DEBUG ] 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215704" 22:15:04 INFO [http_cmd]: http status code OK 22:15:04 DEBUG --- status code --- 22:15:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:15:04 DEBUG --- http response --- 22:15:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:15:04 DEBUG ----- output ----- 22:15:04 DEBUG { 22:15:04 DEBUG "status": "success", 22:15:04 DEBUG "data": { 22:15:04 DEBUG "resultType": "vector", 22:15:04 DEBUG "result": [ 22:15:04 DEBUG { 22:15:04 DEBUG "metric": { 22:15:04 DEBUG "instance": "10.142.0.107:9100" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG [...] 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG 22:15:04 DEBUG "instance": "10.142.0.43:9100" 22:15:04 DEBUG }, 22:15:04 DEBUG "value": [ 22:15:04 DEBUG 1713215704, 22:15:04 DEBUG "13862581248" 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG ] 22:15:04 DEBUG } 22:15:04 DEBUG } 22:15:04 INFO 22:15:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:15:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:15:04 INFO [loop_until]: OK (rc = 0) 22:15:04 DEBUG --- stdout --- 22:15:04 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12906m 8634Mi am-7d597cf7d6-8vr8f 11659m 9066Mi am-7d597cf7d6-9wz9h 15612m 9234Mi ds-cts-0 14682m 8446Mi ds-cts-1 10976m 8501Mi ds-cts-2 10814m 8676Mi ds-idrepo-0 13470m 4956Mi ds-idrepo-1 2074m 2425Mi ds-idrepo-2 2027m 2399Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6647m 2352Mi 22:15:04 DEBUG --- stderr --- 22:15:04 DEBUG 22:15:05 INFO 22:15:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:15:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:15:05 INFO [loop_until]: OK (rc = 0) 22:15:05 DEBUG --- stdout --- 22:15:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5071Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10988m 36% 10722Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13150m 44% 10707Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17565m 58% 3059Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2072m 6% 5234Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16566m 55% 12749Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2955Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12754m 42% 10555Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2087m 6% 4666Mi 4% gke-xlou-cdm-ds-0f601751-j488 10715m 35% 10269Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2443Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17074m 57% 10884Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13888m 46% 7121Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6870m 43% 6211Mi 10% 22:15:05 DEBUG --- stderr --- 22:15:05 DEBUG 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "2297.866666666667" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG { 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG { 22:16:04 DEBUG { 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG "data": { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG "data": { 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG "result": [ 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG { 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG { 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "ds-idrepo-2" 22:16:04 DEBUG "data": { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:16:04 DEBUG "instance": "10.142.0.107:9100" 22:16:04 DEBUG "instance": "10.142.0.107:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:16:04 DEBUG }, 22:16:04 DEBUG }, 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG "result": [ 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG "value": [ 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG { 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "metric": { 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG }, 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG [...] 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG "value": [ 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "ds-cts-2" 22:16:04 DEBUG 22:16:04 ERROR [http_cmd]: ERROR 22:16:04 DEBUG 22:16:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:16:04 DEBUG "instance": "10.142.0.43:9100" 22:16:04 DEBUG "instance": "10.142.0.43:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG [...] 22:16:04 ERROR --- status code --- 22:16:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:16:04 DEBUG }, 22:16:04 DEBUG }, 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG "value": [ 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG 22:16:04 ERROR --- http response --- 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:16:04 DEBUG "2286.3333333333335" 22:16:04 DEBUG "3002.6666666666665" 22:16:04 DEBUG "12306.566666666664" 22:16:04 DEBUG ] 22:16:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:16:04 DEBUG "2564448256" 22:16:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:16:04 DEBUG ] 22:16:04 DEBUG ] 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG }, 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG "value": [ 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG ] 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG "0.079" 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "ds-idrepo-2" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "ds-idrepo-0" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "412967.5" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "ds-idrepo-2" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "ds-idrepo-0" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "instance": "10.142.0.107:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "instance": "10.142.0.43:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0.000500000000000019" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "2297.3333333333335" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "ds-idrepo-2" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "ds-idrepo-0" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "6.712851563908883" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "ds-idrepo-2" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "ds-idrepo-0" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "instance": "10.142.0.107:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "instance": "10.142.0.43:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0.0003333333333333299" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "2291.8" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:16:04 DEBUG 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:16:04 DEBUG { 22:16:04 DEBUG }, 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "value": [ 22:16:04 DEBUG "data": { 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "0" 22:16:04 DEBUG "result": [ 22:16:04 DEBUG ] 22:16:04 DEBUG { 22:16:04 DEBUG } 22:16:04 DEBUG "metric": { 22:16:04 DEBUG ] 22:16:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:16:04 DEBUG } 22:16:04 DEBUG }, 22:16:04 DEBUG } 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "ds-idrepo-2" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "ds-idrepo-0" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "ds-idrepo-2" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "ds-idrepo-0" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "0" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215764" 22:16:04 INFO [http_cmd]: http status code OK 22:16:04 DEBUG --- status code --- 22:16:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:16:04 DEBUG --- http response --- 22:16:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:16:04 DEBUG ----- output ----- 22:16:04 DEBUG { 22:16:04 DEBUG "status": "success", 22:16:04 DEBUG "data": { 22:16:04 DEBUG "resultType": "vector", 22:16:04 DEBUG "result": [ 22:16:04 DEBUG { 22:16:04 DEBUG "metric": { 22:16:04 DEBUG "instance": "10.142.0.107:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG [...] 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG 22:16:04 DEBUG "instance": "10.142.0.43:9100" 22:16:04 DEBUG }, 22:16:04 DEBUG "value": [ 22:16:04 DEBUG 1713215764, 22:16:04 DEBUG "13863195648" 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG ] 22:16:04 DEBUG } 22:16:04 DEBUG } 22:16:04 INFO 22:16:04 INFO [loop_until]: kubectl --namespace=xlou top pods 22:16:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:16:05 INFO [loop_until]: OK (rc = 0) 22:16:05 DEBUG --- stdout --- 22:16:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12912m 8635Mi am-7d597cf7d6-8vr8f 11628m 9064Mi am-7d597cf7d6-9wz9h 15485m 9235Mi ds-cts-0 15699m 8472Mi ds-cts-1 11275m 8503Mi ds-cts-2 10533m 8683Mi ds-idrepo-0 13385m 4955Mi ds-idrepo-1 2022m 2445Mi ds-idrepo-2 2016m 2426Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6771m 2355Mi 22:16:05 DEBUG --- stderr --- 22:16:05 DEBUG 22:16:05 INFO 22:16:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:16:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:16:05 INFO [loop_until]: OK (rc = 0) 22:16:05 DEBUG --- stdout --- 22:16:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5075Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11002m 36% 10715Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13219m 44% 10719Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17541m 58% 3054Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2042m 6% 5273Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16814m 56% 12770Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2959Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12648m 42% 10565Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2092m 7% 4673Mi 4% gke-xlou-cdm-ds-0f601751-j488 10670m 35% 10272Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2443Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17190m 57% 10890Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13816m 46% 7126Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6861m 43% 6210Mi 10% 22:16:05 DEBUG --- stderr --- 22:16:05 DEBUG 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "2259.8" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "ds-idrepo-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "ds-cts-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0.079" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "ds-idrepo-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "ds-idrepo-0" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG { 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "instance": "10.142.0.107:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "instance": "10.142.0.43:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "11930.466666666664" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:17:04 ERROR [http_cmd]: ERROR 22:17:04 ERROR --- status code --- 22:17:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:17:04 ERROR --- http response --- 22:17:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:17:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "instance": "10.142.0.107:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "instance": "10.142.0.43:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "2727.5333333333333" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "2275.6" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "2566164480" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:17:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:17:04 DEBUG }, 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG "413104.5" 22:17:04 DEBUG ] 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "instance": "10.142.0.107:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "instance": "10.142.0.43:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0.00033333333333350387" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "ds-idrepo-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "ds-idrepo-0" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "410.5031068350371" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "2261.133333333333" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "ds-idrepo-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "ds-idrepo-0" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "ds-idrepo-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "ds-idrepo-0" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "6.700478272430529" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "instance": "10.142.0.107:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "instance": "10.142.0.43:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0.00033333333333333365" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "2285.9333333333334" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "idm-b85f9d978-j5pbm" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "ds-idrepo-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "ds-idrepo-0" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "ds-idrepo-2" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "ds-idrepo-0" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "0" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215824" 22:17:04 INFO [http_cmd]: http status code OK 22:17:04 DEBUG --- status code --- 22:17:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:17:04 DEBUG --- http response --- 22:17:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:17:04 DEBUG ----- output ----- 22:17:04 DEBUG { 22:17:04 DEBUG "status": "success", 22:17:04 DEBUG "data": { 22:17:04 DEBUG "resultType": "vector", 22:17:04 DEBUG "result": [ 22:17:04 DEBUG { 22:17:04 DEBUG "metric": { 22:17:04 DEBUG "instance": "10.142.0.107:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG [...] 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG 22:17:04 DEBUG "instance": "10.142.0.43:9100" 22:17:04 DEBUG }, 22:17:04 DEBUG "value": [ 22:17:04 DEBUG 1713215824, 22:17:04 DEBUG "13863879680" 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG ] 22:17:04 DEBUG } 22:17:04 DEBUG } 22:17:05 INFO 22:17:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:17:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:17:05 INFO [loop_until]: OK (rc = 0) 22:17:05 DEBUG --- stdout --- 22:17:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12577m 8690Mi am-7d597cf7d6-8vr8f 11682m 9070Mi am-7d597cf7d6-9wz9h 16300m 9231Mi ds-cts-0 15316m 8454Mi ds-cts-1 11204m 8499Mi ds-cts-2 10524m 8691Mi ds-idrepo-0 13522m 4955Mi ds-idrepo-1 2001m 2503Mi ds-idrepo-2 1986m 2428Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 6m 1026Mi idm-b85f9d978-rl76f 9m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6786m 2357Mi 22:17:05 DEBUG --- stderr --- 22:17:05 DEBUG 22:17:05 INFO 22:17:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:17:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:17:05 INFO [loop_until]: OK (rc = 0) 22:17:05 DEBUG --- stdout --- 22:17:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5086Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11023m 36% 10724Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13185m 44% 10767Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17490m 58% 3060Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2065m 6% 5280Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15875m 53% 12758Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2958Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12659m 42% 10570Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2081m 6% 4730Mi 4% gke-xlou-cdm-ds-0f601751-j488 10686m 35% 10278Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2451Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17278m 57% 10911Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13792m 46% 7124Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6878m 43% 6213Mi 10% 22:17:05 DEBUG --- stderr --- 22:17:05 DEBUG 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-idrepo-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "ds-cts-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "2290.4333333333334" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0.079" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-idrepo-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "ds-idrepo-0" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "412846.5" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "instance": "10.142.0.107:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "instance": "10.142.0.43:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "12011.166666666664" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215884" 22:18:04 ERROR [http_cmd]: ERROR 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 ERROR --- status code --- 22:18:04 DEBUG --- status code --- 22:18:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 ERROR --- http response --- 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:18:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:18:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG { 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "data": { 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:18:04 DEBUG "result": [ 22:18:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:18:04 DEBUG }, 22:18:04 DEBUG { 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG "instance": "10.142.0.107:9100" 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG }, 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG }, 22:18:04 DEBUG 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "instance": "10.142.0.43:9100" 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "2281.7666666666664" 22:18:04 DEBUG }, 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG "value": [ 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG } 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG ] 22:18:04 DEBUG "3508.1666666666665" 22:18:04 DEBUG ] 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG { 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG ] 22:18:04 DEBUG "data": { 22:18:04 DEBUG } 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG } 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "2568810496" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "instance": "10.142.0.107:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "instance": "10.142.0.43:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0.0005333333333335114" 22:18:04 DEBUG ] 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-idrepo-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "ds-idrepo-0" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG "result": [ 22:18:04 DEBUG "result": [ 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG { 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:18:04 DEBUG "pod": "ds-cts-1" 22:18:04 DEBUG "data": { 22:18:04 DEBUG }, 22:18:04 DEBUG }, 22:18:04 DEBUG "resultType": "vector", 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 DEBUG "value": [ 22:18:04 DEBUG "value": [ 22:18:04 DEBUG "result": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG { 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "metric": { 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:18:04 DEBUG [...] 22:18:04 DEBUG [...] 22:18:04 DEBUG }, 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:18:04 DEBUG 22:18:04 DEBUG }, 22:18:04 DEBUG }, 22:18:04 DEBUG [...] 22:18:04 DEBUG "value": [ 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG 22:18:04 DEBUG "2290.1666666666665" 22:18:04 DEBUG "0" 22:18:04 DEBUG 22:18:04 DEBUG ] 22:18:04 DEBUG ] 22:18:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG }, 22:18:04 DEBUG ] 22:18:04 DEBUG ] 22:18:04 DEBUG "value": [ 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG "6.698266507973827" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-idrepo-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "ds-idrepo-0" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "2292.9666666666667" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-idrepo-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "ds-idrepo-0" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-cts-1" 22:18:04 DEBUG "pod": "ds-cts-1" 22:18:04 DEBUG }, 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:18:04 DEBUG }, 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "instance": "10.142.0.107:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "instance": "10.142.0.43:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-idrepo-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "ds-idrepo-0" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215884" 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "instance": "10.142.0.107:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "instance": "10.142.0.43:9100" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "13864557568" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-idrepo-2" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "ds-idrepo-0" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:04 INFO [http_cmd]: http status code OK 22:18:04 DEBUG --- status code --- 22:18:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:18:04 DEBUG --- http response --- 22:18:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:18:04 DEBUG ----- output ----- 22:18:04 DEBUG { 22:18:04 DEBUG "status": "success", 22:18:04 DEBUG "data": { 22:18:04 DEBUG "resultType": "vector", 22:18:04 DEBUG "result": [ 22:18:04 DEBUG { 22:18:04 DEBUG "metric": { 22:18:04 DEBUG "pod": "ds-cts-1" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG [...] 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG 22:18:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:18:04 DEBUG }, 22:18:04 DEBUG "value": [ 22:18:04 DEBUG 1713215884, 22:18:04 DEBUG "0" 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG ] 22:18:04 DEBUG } 22:18:04 DEBUG } 22:18:05 INFO 22:18:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:18:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:18:05 INFO [loop_until]: OK (rc = 0) 22:18:05 DEBUG --- stdout --- 22:18:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12971m 8645Mi am-7d597cf7d6-8vr8f 11961m 9069Mi am-7d597cf7d6-9wz9h 15995m 9270Mi ds-cts-0 15313m 8464Mi ds-cts-1 10658m 8520Mi ds-cts-2 10817m 8696Mi ds-idrepo-0 13572m 4955Mi ds-idrepo-1 1974m 2503Mi ds-idrepo-2 2020m 2462Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 3m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6540m 2359Mi 22:18:05 DEBUG --- stderr --- 22:18:05 DEBUG 22:18:05 INFO 22:18:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:18:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:18:05 INFO [loop_until]: OK (rc = 0) 22:18:05 DEBUG --- stdout --- 22:18:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5071Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10483m 35% 10735Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13153m 44% 10734Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17540m 58% 3053Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2041m 6% 5300Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16433m 55% 12760Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2963Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12659m 42% 10569Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2079m 6% 4715Mi 4% gke-xlou-cdm-ds-0f601751-j488 10971m 36% 10291Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16625m 55% 10870Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13942m 46% 7123Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6853m 43% 6214Mi 10% 22:18:05 DEBUG --- stderr --- 22:18:05 DEBUG 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "ds-idrepo-2" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "ds-cts-2" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "ds-idrepo-2" 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:19:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:19:04 DEBUG }, 22:19:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:19:04 DEBUG }, 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "ds-idrepo-0" 22:19:04 DEBUG 22:19:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:19:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:19:04 DEBUG }, 22:19:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:19:04 DEBUG }, 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0.079" 22:19:04 DEBUG "413953" 22:19:04 DEBUG ] 22:19:04 DEBUG "2305.0666666666666" 22:19:04 DEBUG ] 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215944" 22:19:04 ERROR [http_cmd]: ERROR 22:19:04 ERROR --- status code --- 22:19:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:19:04 ERROR --- http response --- 22:19:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:19:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "instance": "10.142.0.107:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "instance": "10.142.0.43:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "13041.23333333333" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG } 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "2294" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "instance": "10.142.0.107:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "instance": "10.142.0.43:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "5106.366666666666" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "2568441856" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "instance": "10.142.0.107:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "instance": "10.142.0.43:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0.0002666666666669926" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "ds-idrepo-2" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "ds-idrepo-0" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "6.729024188148233" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "2305.5" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "ds-idrepo-2" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "ds-idrepo-0" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "2311.0333333333333" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "ds-idrepo-2" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "ds-idrepo-0" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "instance": "10.142.0.107:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "instance": "10.142.0.43:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0.00002068965517242825" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "ds-idrepo-2" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "ds-idrepo-0" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713215944" 22:19:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713215944" 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "instance": "10.142.0.107:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "instance": "10.142.0.43:9100" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "13865380864" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:04 INFO [http_cmd]: http status code OK 22:19:04 DEBUG --- status code --- 22:19:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:19:04 DEBUG --- http response --- 22:19:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:19:04 DEBUG ----- output ----- 22:19:04 DEBUG { 22:19:04 DEBUG "status": "success", 22:19:04 DEBUG "data": { 22:19:04 DEBUG "resultType": "vector", 22:19:04 DEBUG "result": [ 22:19:04 DEBUG { 22:19:04 DEBUG "metric": { 22:19:04 DEBUG "pod": "ds-idrepo-2" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG [...] 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG 22:19:04 DEBUG "pod": "ds-idrepo-0" 22:19:04 DEBUG }, 22:19:04 DEBUG "value": [ 22:19:04 DEBUG 1713215944, 22:19:04 DEBUG "0" 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG ] 22:19:04 DEBUG } 22:19:04 DEBUG } 22:19:05 INFO 22:19:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:19:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:19:05 INFO [loop_until]: OK (rc = 0) 22:19:05 DEBUG --- stdout --- 22:19:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12373m 8665Mi am-7d597cf7d6-8vr8f 11857m 9068Mi am-7d597cf7d6-9wz9h 16130m 9235Mi ds-cts-0 15365m 8460Mi ds-cts-1 11074m 8516Mi ds-cts-2 10684m 8692Mi ds-idrepo-0 13333m 4955Mi ds-idrepo-1 2057m 2504Mi ds-idrepo-2 1961m 2462Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6586m 2361Mi 22:19:05 DEBUG --- stderr --- 22:19:05 DEBUG 22:19:05 INFO 22:19:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:19:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:19:05 INFO [loop_until]: OK (rc = 0) 22:19:05 DEBUG --- stdout --- 22:19:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5077Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11018m 36% 10752Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13245m 44% 10752Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17600m 58% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2023m 6% 5307Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16609m 55% 12763Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2950Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12591m 42% 10564Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2094m 7% 4727Mi 4% gke-xlou-cdm-ds-0f601751-j488 10642m 35% 10306Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2451Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17326m 58% 10886Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13968m 46% 7130Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6865m 43% 6213Mi 10% 22:19:05 DEBUG --- stderr --- 22:19:05 DEBUG 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-idrepo-2" 22:20:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:20:04 DEBUG }, 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "ds-cts-2" 22:20:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:20:04 DEBUG }, 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG "2277" 22:20:04 DEBUG ] 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0.08" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "414118" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "instance": "10.142.0.107:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "instance": "10.142.0.43:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "12918.399999999998" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-idrepo-2" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "ds-idrepo-0" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216004" 22:20:04 ERROR [http_cmd]: ERROR 22:20:04 ERROR --- status code --- 22:20:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:20:04 ERROR --- http response --- 22:20:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:20:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "2573799424" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "instance": "10.142.0.107:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "instance": "10.142.0.43:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0.000966666666667019" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "2265.0666666666666" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "instance": "10.142.0.107:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "instance": "10.142.0.43:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "3472.333333333333" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-idrepo-2" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "ds-idrepo-0" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "6.777816657269767" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "2277.2666666666664" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-cts-1" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-idrepo-2" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "ds-idrepo-0" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-cts-1" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "2280.633333333333" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-idrepo-2" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "ds-idrepo-0" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216004" 22:20:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216004" 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-cts-1" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "4226.194042269928" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "pod": "ds-cts-1" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0.00002363121983690352" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:04 INFO [http_cmd]: http status code OK 22:20:04 DEBUG --- status code --- 22:20:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:04 DEBUG --- http response --- 22:20:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:04 DEBUG ----- output ----- 22:20:04 DEBUG { 22:20:04 DEBUG "status": "success", 22:20:04 DEBUG "data": { 22:20:04 DEBUG "resultType": "vector", 22:20:04 DEBUG "result": [ 22:20:04 DEBUG { 22:20:04 DEBUG "metric": { 22:20:04 DEBUG "instance": "10.142.0.107:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG [...] 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG 22:20:04 DEBUG "instance": "10.142.0.43:9100" 22:20:04 DEBUG }, 22:20:04 DEBUG "value": [ 22:20:04 DEBUG 1713216004, 22:20:04 DEBUG "0.00033333333333333365" 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG ] 22:20:04 DEBUG } 22:20:04 DEBUG } 22:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216005" 22:20:05 INFO [http_cmd]: http status code OK 22:20:05 DEBUG --- status code --- 22:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:05 DEBUG --- http response --- 22:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:20:05 DEBUG ----- output ----- 22:20:05 DEBUG { 22:20:05 DEBUG "status": "success", 22:20:05 DEBUG "data": { 22:20:05 DEBUG "resultType": "vector", 22:20:05 DEBUG "result": [ 22:20:05 DEBUG { 22:20:05 DEBUG "metric": { 22:20:05 DEBUG "pod": "ds-idrepo-2" 22:20:05 DEBUG }, 22:20:05 DEBUG "value": [ 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG [...] 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG "pod": "ds-idrepo-0" 22:20:05 DEBUG }, 22:20:05 DEBUG "value": [ 22:20:05 DEBUG 1713216005, 22:20:05 DEBUG "0" 22:20:05 DEBUG ] 22:20:05 DEBUG } 22:20:05 DEBUG ] 22:20:05 DEBUG } 22:20:05 DEBUG } 22:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216005" 22:20:05 INFO [http_cmd]: http status code OK 22:20:05 DEBUG --- status code --- 22:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:05 DEBUG --- http response --- 22:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:20:05 DEBUG ----- output ----- 22:20:05 DEBUG { 22:20:05 DEBUG "status": "success", 22:20:05 DEBUG "data": { 22:20:05 DEBUG "resultType": "vector", 22:20:05 DEBUG "result": [ 22:20:05 DEBUG { 22:20:05 DEBUG "metric": { 22:20:05 DEBUG "pod": "ds-idrepo-2" 22:20:05 DEBUG }, 22:20:05 DEBUG "value": [ 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG [...] 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG "pod": "ds-idrepo-0" 22:20:05 DEBUG }, 22:20:05 DEBUG "value": [ 22:20:05 DEBUG 1713216005, 22:20:05 DEBUG "0" 22:20:05 DEBUG ] 22:20:05 DEBUG } 22:20:05 DEBUG ] 22:20:05 DEBUG } 22:20:05 DEBUG } 22:20:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216005" 22:20:05 INFO [http_cmd]: http status code OK 22:20:05 DEBUG --- status code --- 22:20:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:20:05 DEBUG --- http response --- 22:20:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:20:05 DEBUG ----- output ----- 22:20:05 DEBUG { 22:20:05 DEBUG "status": "success", 22:20:05 DEBUG "data": { 22:20:05 DEBUG "resultType": "vector", 22:20:05 DEBUG "result": [ 22:20:05 DEBUG { 22:20:05 DEBUG "metric": { 22:20:05 DEBUG "instance": "10.142.0.107:9100" 22:20:05 DEBUG }, 22:20:05 DEBUG "value": [ 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG [...] 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG 22:20:05 DEBUG "instance": "10.142.0.43:9100" 22:20:05 DEBUG }, 22:20:05 DEBUG "value": [ 22:20:05 DEBUG 1713216005, 22:20:05 DEBUG "13865970688" 22:20:05 DEBUG ] 22:20:05 DEBUG } 22:20:05 DEBUG ] 22:20:05 DEBUG } 22:20:05 DEBUG } 22:20:05 INFO 22:20:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:20:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:20:05 INFO [loop_until]: OK (rc = 0) 22:20:05 DEBUG --- stdout --- 22:20:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12804m 8664Mi am-7d597cf7d6-8vr8f 11690m 9069Mi am-7d597cf7d6-9wz9h 15575m 9234Mi ds-cts-0 15544m 8460Mi ds-cts-1 11524m 8510Mi ds-cts-2 10848m 8699Mi ds-idrepo-0 13331m 4955Mi ds-idrepo-1 2022m 2504Mi ds-idrepo-2 1950m 2487Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 4m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6725m 2363Mi 22:20:05 DEBUG --- stderr --- 22:20:05 DEBUG 22:20:05 INFO 22:20:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:20:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:20:05 INFO [loop_until]: OK (rc = 0) 22:20:05 DEBUG --- stdout --- 22:20:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5080Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10879m 36% 10735Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13256m 44% 10750Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17579m 58% 3071Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2051m 6% 5330Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16300m 54% 12793Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2952Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12753m 42% 10574Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2106m 7% 4726Mi 4% gke-xlou-cdm-ds-0f601751-j488 10269m 34% 10301Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 59m 0% 2451Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17040m 57% 10902Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13454m 45% 7121Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6928m 43% 6209Mi 10% 22:20:05 DEBUG --- stderr --- 22:20:05 DEBUG 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "0.08" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "ds-idrepo-2" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "ds-cts-2" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "0" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "2283.2" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "413366" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216064" 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "instance": "10.142.0.107:9100" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "instance": "10.142.0.43:9100" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "11733.799999999997" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "ds-idrepo-2" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "ds-idrepo-0" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "0" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:21:04 ERROR [http_cmd]: ERROR 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 ERROR --- status code --- 22:21:04 DEBUG --- status code --- 22:21:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 ERROR --- http response --- 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:21:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "0" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216064" 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "2575605760" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "instance": "10.142.0.107:9100" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "instance": "10.142.0.43:9100" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "0.00030000000000001136" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "ds-idrepo-2" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "ds-idrepo-0" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "0" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "instance": "10.142.0.107:9100" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "instance": "10.142.0.43:9100" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "4776.866666666666" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "2293.0666666666666" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216064" 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "2284.8" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: http status code OK 22:21:04 DEBUG --- status code --- 22:21:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:04 DEBUG --- http response --- 22:21:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:04 DEBUG ----- output ----- 22:21:04 DEBUG { 22:21:04 DEBUG "status": "success", 22:21:04 DEBUG "data": { 22:21:04 DEBUG "resultType": "vector", 22:21:04 DEBUG "result": [ 22:21:04 DEBUG { 22:21:04 DEBUG "metric": { 22:21:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG [...] 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG 22:21:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:21:04 DEBUG }, 22:21:04 DEBUG "value": [ 22:21:04 DEBUG 1713216064, 22:21:04 DEBUG "6.608711465777324" 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG ] 22:21:04 DEBUG } 22:21:04 DEBUG } 22:21:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216064" 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216064, 22:21:05 DEBUG "7946699.15214238" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "ds-idrepo-2" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "ds-idrepo-0" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "0" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216065" 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:21:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:21:05 DEBUG }, 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:05 DEBUG }, 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "0" 22:21:05 DEBUG "2281.2" 22:21:05 DEBUG ] 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "ds-idrepo-2" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "ds-idrepo-0" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "0" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "instance": "10.142.0.107:9100" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "instance": "10.142.0.43:9100" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "0" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "258447112.75579074" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "ds-idrepo-2" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "ds-idrepo-0" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "0" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "10.854008419115226" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216065" 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "pod": "ds-idrepo-2" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "pod": "ds-idrepo-0" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "0" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216065" 22:21:05 INFO [http_cmd]: http status code OK 22:21:05 DEBUG --- status code --- 22:21:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:21:05 DEBUG --- http response --- 22:21:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:21:05 DEBUG ----- output ----- 22:21:05 DEBUG { 22:21:05 DEBUG "status": "success", 22:21:05 DEBUG "data": { 22:21:05 DEBUG "resultType": "vector", 22:21:05 DEBUG "result": [ 22:21:05 DEBUG { 22:21:05 DEBUG "metric": { 22:21:05 DEBUG "instance": "10.142.0.107:9100" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG [...] 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG 22:21:05 DEBUG "instance": "10.142.0.43:9100" 22:21:05 DEBUG }, 22:21:05 DEBUG "value": [ 22:21:05 DEBUG 1713216065, 22:21:05 DEBUG "13866683392" 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG ] 22:21:05 DEBUG } 22:21:05 DEBUG } 22:21:05 INFO 22:21:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:21:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:21:05 INFO [loop_until]: OK (rc = 0) 22:21:05 DEBUG --- stdout --- 22:21:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12448m 8678Mi am-7d597cf7d6-8vr8f 11763m 9070Mi am-7d597cf7d6-9wz9h 16251m 9232Mi ds-cts-0 15394m 8465Mi ds-cts-1 11259m 8523Mi ds-cts-2 11688m 8698Mi ds-idrepo-0 13709m 4955Mi ds-idrepo-1 1956m 2531Mi ds-idrepo-2 1964m 2489Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6523m 2363Mi 22:21:05 DEBUG --- stderr --- 22:21:05 DEBUG 22:21:05 INFO 22:21:05 INFO [loop_until]: kubectl --namespace=xlou top node 22:21:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:21:05 INFO [loop_until]: OK (rc = 0) 22:21:05 DEBUG --- stdout --- 22:21:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5083Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11002m 36% 10752Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13370m 44% 10758Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17529m 58% 3069Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2046m 6% 5337Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16552m 55% 12776Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2961Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12717m 42% 10562Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2100m 7% 4758Mi 4% gke-xlou-cdm-ds-0f601751-j488 10876m 36% 10315Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17142m 57% 10891Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14038m 47% 7125Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6754m 42% 6213Mi 10% 22:21:05 DEBUG --- stderr --- 22:21:05 DEBUG 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "ds-idrepo-2" 22:22:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:22:04 DEBUG }, 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "ds-cts-2" 22:22:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:22:04 DEBUG }, 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "0" 22:22:04 DEBUG "0.08" 22:22:04 DEBUG ] 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "2301.7" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "412484" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216124" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "instance": "10.142.0.107:9100" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "instance": "10.142.0.43:9100" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "13298.133333333333" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "ds-idrepo-2" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "ds-idrepo-0" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "0" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:22:04 ERROR [http_cmd]: ERROR 22:22:04 ERROR --- status code --- 22:22:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:22:04 ERROR --- http response --- 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:22:04 DEBUG --- status code --- 22:22:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "0" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216124" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "2576662528" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "instance": "10.142.0.107:9100" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "instance": "10.142.0.43:9100" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "0.0005333333333335114" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "ds-idrepo-2" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "ds-idrepo-0" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "0" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216124" 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216124" 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "2302.3333333333335" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 INFO [http_cmd]: http status code OK 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG --- status code --- 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG --- http response --- 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "instance": "10.142.0.107:9100" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "instance": "10.142.0.43:9100" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "3224.7666666666664" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:04 DEBUG ----- output ----- 22:22:04 DEBUG { 22:22:04 DEBUG "status": "success", 22:22:04 DEBUG "data": { 22:22:04 DEBUG "resultType": "vector", 22:22:04 DEBUG "result": [ 22:22:04 DEBUG { 22:22:04 DEBUG "metric": { 22:22:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG [...] 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG 22:22:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:22:04 DEBUG }, 22:22:04 DEBUG "value": [ 22:22:04 DEBUG 1713216124, 22:22:04 DEBUG "2304.8" 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG ] 22:22:04 DEBUG } 22:22:04 DEBUG } 22:22:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216124" 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216124, 22:22:05 DEBUG "6.588370195335446" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "pod": "ds-cts-1" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "pod": "ds-idrepo-2" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "pod": "ds-idrepo-0" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0" 22:22:05 DEBUG ] 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:22:05 DEBUG } 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG ] 22:22:05 DEBUG { 22:22:05 DEBUG } 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG } 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "pod": "ds-idrepo-2" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "pod": "ds-idrepo-0" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "2293.1666666666665" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216125" 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG { 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG "data": { 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:22:05 DEBUG "metric": { 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG "pod": "ds-idrepo-2" 22:22:05 DEBUG { 22:22:05 DEBUG }, 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "value": [ 22:22:05 DEBUG "data": { 22:22:05 DEBUG 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG 22:22:05 DEBUG "result": [ 22:22:05 DEBUG 22:22:05 DEBUG { 22:22:05 DEBUG [...] 22:22:05 DEBUG "metric": { 22:22:05 DEBUG 22:22:05 DEBUG "pod": "ds-cts-1" 22:22:05 DEBUG 22:22:05 DEBUG }, 22:22:05 DEBUG 22:22:05 DEBUG "value": [ 22:22:05 DEBUG "pod": "ds-idrepo-0" 22:22:05 DEBUG 22:22:05 DEBUG }, 22:22:05 DEBUG 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG [...] 22:22:05 DEBUG "0" 22:22:05 DEBUG 22:22:05 DEBUG ] 22:22:05 DEBUG 22:22:05 DEBUG } 22:22:05 DEBUG 22:22:05 DEBUG ] 22:22:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:22:05 DEBUG } 22:22:05 DEBUG }, 22:22:05 DEBUG } 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "instance": "10.142.0.107:9100" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "instance": "10.142.0.43:9100" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "pod": "ds-cts-1" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "pod": "ds-cts-1" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0.000019396743482514234" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216125" 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "pod": "ds-idrepo-2" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "pod": "ds-idrepo-0" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "0" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216125" 22:22:05 INFO [http_cmd]: http status code OK 22:22:05 DEBUG --- status code --- 22:22:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:22:05 DEBUG --- http response --- 22:22:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:22:05 DEBUG ----- output ----- 22:22:05 DEBUG { 22:22:05 DEBUG "status": "success", 22:22:05 DEBUG "data": { 22:22:05 DEBUG "resultType": "vector", 22:22:05 DEBUG "result": [ 22:22:05 DEBUG { 22:22:05 DEBUG "metric": { 22:22:05 DEBUG "instance": "10.142.0.107:9100" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG [...] 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG 22:22:05 DEBUG "instance": "10.142.0.43:9100" 22:22:05 DEBUG }, 22:22:05 DEBUG "value": [ 22:22:05 DEBUG 1713216125, 22:22:05 DEBUG "13867340800" 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG ] 22:22:05 DEBUG } 22:22:05 DEBUG } 22:22:05 INFO 22:22:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:22:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:22:05 INFO [loop_until]: OK (rc = 0) 22:22:05 DEBUG --- stdout --- 22:22:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12701m 8666Mi am-7d597cf7d6-8vr8f 11500m 9068Mi am-7d597cf7d6-9wz9h 16001m 9231Mi ds-cts-0 15536m 8475Mi ds-cts-1 11073m 8532Mi ds-cts-2 10741m 8714Mi ds-idrepo-0 13646m 4955Mi ds-idrepo-1 1984m 2531Mi ds-idrepo-2 1985m 2506Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6750m 2365Mi 22:22:05 DEBUG --- stderr --- 22:22:05 DEBUG 22:22:06 INFO 22:22:06 INFO [loop_until]: kubectl --namespace=xlou top node 22:22:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:22:06 INFO [loop_until]: OK (rc = 0) 22:22:06 DEBUG --- stdout --- 22:22:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5079Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11057m 37% 10766Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13265m 44% 10747Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17602m 58% 3071Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2035m 6% 5356Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16607m 55% 12777Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2948Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12689m 42% 10556Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2084m 6% 4749Mi 4% gke-xlou-cdm-ds-0f601751-j488 10473m 35% 10312Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2460Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17194m 57% 10886Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14016m 46% 7126Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6748m 42% 6211Mi 10% 22:22:06 DEBUG --- stderr --- 22:22:06 DEBUG 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1713216184, 22:23:04 DEBUG "2296.7" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216184" 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1713216184, 22:23:04 DEBUG "412312.5" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "pod": "ds-idrepo-2" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "pod": "ds-cts-2" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1713216184, 22:23:04 DEBUG "0" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG }, 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG "value": [ 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:04 DEBUG [...] 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG 22:23:04 DEBUG { 22:23:04 DEBUG 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG 22:23:04 DEBUG "data": { 22:23:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG }, 22:23:04 DEBUG "result": [ 22:23:04 DEBUG "value": [ 22:23:04 DEBUG { 22:23:04 DEBUG 1713216184, 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "0.08" 22:23:04 DEBUG "instance": "10.142.0.107:9100" 22:23:04 DEBUG ] 22:23:04 DEBUG }, 22:23:04 DEBUG } 22:23:04 DEBUG "value": [ 22:23:04 DEBUG ] 22:23:04 DEBUG 22:23:04 DEBUG } 22:23:04 DEBUG 22:23:04 DEBUG } 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "instance": "10.142.0.43:9100" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1713216184, 22:23:04 DEBUG "12817.399999999998" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216184" 22:23:04 ERROR [http_cmd]: ERROR 22:23:04 ERROR --- status code --- 22:23:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:23:04 ERROR --- http response --- 22:23:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:23:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "pod": "ds-idrepo-2" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "pod": "ds-idrepo-0" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1713216184, 22:23:04 DEBUG "0" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:23:04 INFO [http_cmd]: http status code OK 22:23:04 DEBUG --- status code --- 22:23:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:04 DEBUG --- http response --- 22:23:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:04 DEBUG ----- output ----- 22:23:04 DEBUG { 22:23:04 DEBUG "status": "success", 22:23:04 DEBUG "data": { 22:23:04 DEBUG "resultType": "vector", 22:23:04 DEBUG "result": [ 22:23:04 DEBUG { 22:23:04 DEBUG "metric": { 22:23:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG [...] 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG 22:23:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:04 DEBUG }, 22:23:04 DEBUG "value": [ 22:23:04 DEBUG 1713216184, 22:23:04 DEBUG "0" 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG ] 22:23:04 DEBUG } 22:23:04 DEBUG } 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216184" 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216184" 22:23:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216184" 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216184, 22:23:05 DEBUG "2578460672" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216184, 22:23:05 DEBUG "2298.1666666666665" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "instance": "10.142.0.107:9100" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "instance": "10.142.0.43:9100" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216184, 22:23:05 DEBUG "0.00043333333333303397" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-idrepo-2" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "ds-idrepo-0" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "0" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "2275.5333333333333" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216185" 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG { 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "instance": "10.142.0.107:9100" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "instance": "10.142.0.43:9100" 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:05 DEBUG }, 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG "value": [ 22:23:05 DEBUG { 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "3622.4333333333334" 22:23:05 DEBUG "data": { 22:23:05 DEBUG ] 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG } 22:23:05 DEBUG "result": [ 22:23:05 DEBUG ] 22:23:05 DEBUG { 22:23:05 DEBUG } 22:23:05 DEBUG "metric": { 22:23:05 DEBUG } 22:23:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "6.640571268595573" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-idrepo-2" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "ds-idrepo-0" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "0" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "2300.9" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-idrepo-2" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "ds-idrepo-0" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "0" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-cts-1" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "11207033.434863286" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-idrepo-2" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "ds-idrepo-0" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG }, 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG "value": [ 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG "0" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:23:05 DEBUG } 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG } 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-cts-1" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "0" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "instance": "10.142.0.107:9100" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "instance": "10.142.0.43:9100" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "0" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-idrepo-2" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "ds-idrepo-0" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "0" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216185" 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-cts-1" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "338987988.65665376" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "pod": "ds-cts-1" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "15.944892343239038" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216185" 22:23:05 INFO [http_cmd]: http status code OK 22:23:05 DEBUG --- status code --- 22:23:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:23:05 DEBUG --- http response --- 22:23:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:23:05 DEBUG ----- output ----- 22:23:05 DEBUG { 22:23:05 DEBUG "status": "success", 22:23:05 DEBUG "data": { 22:23:05 DEBUG "resultType": "vector", 22:23:05 DEBUG "result": [ 22:23:05 DEBUG { 22:23:05 DEBUG "metric": { 22:23:05 DEBUG "instance": "10.142.0.107:9100" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG [...] 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG 22:23:05 DEBUG "instance": "10.142.0.43:9100" 22:23:05 DEBUG }, 22:23:05 DEBUG "value": [ 22:23:05 DEBUG 1713216185, 22:23:05 DEBUG "13867983872" 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG ] 22:23:05 DEBUG } 22:23:05 DEBUG } 22:23:05 INFO 22:23:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:23:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:23:05 INFO [loop_until]: OK (rc = 0) 22:23:05 DEBUG --- stdout --- 22:23:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12624m 8668Mi am-7d597cf7d6-8vr8f 12084m 9071Mi am-7d597cf7d6-9wz9h 16207m 9238Mi ds-cts-0 15022m 8475Mi ds-cts-1 11090m 8527Mi ds-cts-2 10871m 8693Mi ds-idrepo-0 13743m 4955Mi ds-idrepo-1 1958m 2531Mi ds-idrepo-2 1964m 2507Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6466m 2367Mi 22:23:05 DEBUG --- stderr --- 22:23:05 DEBUG 22:23:06 INFO 22:23:06 INFO [loop_until]: kubectl --namespace=xlou top node 22:23:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:23:06 INFO [loop_until]: OK (rc = 0) 22:23:06 DEBUG --- stdout --- 22:23:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5079Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11028m 36% 10761Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13163m 44% 10752Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17629m 59% 3068Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2043m 6% 5354Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16304m 54% 12780Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2953Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12758m 42% 10564Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2092m 7% 4761Mi 4% gke-xlou-cdm-ds-0f601751-j488 10789m 36% 10303Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2467Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17285m 57% 10882Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13948m 46% 7128Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6739m 42% 6214Mi 10% 22:23:06 DEBUG --- stderr --- 22:23:06 DEBUG 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216244" 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 DEBUG --- status code --- 22:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:04 DEBUG ----- output ----- 22:24:04 DEBUG { 22:24:04 DEBUG "status": "success", 22:24:04 DEBUG "data": { 22:24:04 DEBUG "resultType": "vector", 22:24:04 DEBUG "result": [ 22:24:04 DEBUG { 22:24:04 DEBUG "metric": { 22:24:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG [...] 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 1713216244, 22:24:04 DEBUG "2307.8" 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG } 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216244" 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216244" 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216244" 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216244" 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216244" 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216244" 22:24:04 DEBUG --- status code --- 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:04 DEBUG --- status code --- 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:24:04 DEBUG ----- output ----- 22:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:04 DEBUG { 22:24:04 DEBUG ----- output ----- 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 DEBUG "status": "success", 22:24:04 DEBUG { 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 DEBUG --- status code --- 22:24:04 DEBUG "data": { 22:24:04 DEBUG "status": "success", 22:24:04 DEBUG --- status code --- 22:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:04 DEBUG "resultType": "vector", 22:24:04 DEBUG "data": { 22:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG "result": [ 22:24:04 DEBUG "resultType": "vector", 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG { 22:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:04 DEBUG "result": [ 22:24:04 DEBUG "metric": { 22:24:04 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:04 DEBUG ----- output ----- 22:24:04 DEBUG { 22:24:04 DEBUG "pod": "ds-idrepo-2" 22:24:04 DEBUG ----- output ----- 22:24:04 DEBUG { 22:24:04 DEBUG "metric": { 22:24:04 DEBUG }, 22:24:04 DEBUG { 22:24:04 DEBUG "status": "success", 22:24:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:24:04 DEBUG "value": [ 22:24:04 DEBUG "status": "success", 22:24:04 DEBUG "data": { 22:24:04 DEBUG }, 22:24:04 DEBUG 22:24:04 DEBUG "data": { 22:24:04 DEBUG "resultType": "vector", 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 22:24:04 DEBUG "resultType": "vector", 22:24:04 DEBUG "result": [ 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG "result": [ 22:24:04 DEBUG { 22:24:04 DEBUG 22:24:04 DEBUG [...] 22:24:04 DEBUG { 22:24:04 DEBUG "metric": { 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG "metric": { 22:24:04 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:24:04 DEBUG [...] 22:24:04 DEBUG 22:24:04 DEBUG "instance": "10.142.0.107:9100" 22:24:04 DEBUG }, 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 22:24:04 DEBUG "pod": "ds-cts-2" 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG }, 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG }, 22:24:04 DEBUG 1713216244, 22:24:04 DEBUG 22:24:04 DEBUG [...] 22:24:04 DEBUG "value": [ 22:24:04 DEBUG "0" 22:24:04 DEBUG [...] 22:24:04 DEBUG 22:24:04 DEBUG 1713216244, 22:24:04 DEBUG ] 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG "0.08" 22:24:04 DEBUG } 22:24:04 DEBUG 22:24:04 DEBUG 22:24:04 DEBUG ] 22:24:04 DEBUG ] 22:24:04 DEBUG 22:24:04 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:04 DEBUG } 22:24:04 DEBUG } 22:24:04 DEBUG "instance": "10.142.0.43:9100" 22:24:04 DEBUG }, 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG }, 22:24:04 DEBUG "value": [ 22:24:04 DEBUG } 22:24:04 DEBUG "value": [ 22:24:04 DEBUG 1713216244, 22:24:04 DEBUG } 22:24:04 DEBUG 1713216244, 22:24:04 DEBUG "412704" 22:24:04 DEBUG "11713.166666666664" 22:24:04 DEBUG ] 22:24:04 DEBUG ] 22:24:04 DEBUG } 22:24:04 DEBUG } 22:24:04 ERROR [http_cmd]: ERROR 22:24:04 DEBUG ] 22:24:04 INFO [http_cmd]: http status code OK 22:24:04 DEBUG ] 22:24:04 ERROR --- status code --- 22:24:04 DEBUG } 22:24:04 DEBUG --- status code --- 22:24:04 DEBUG } 22:24:04 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:24:04 DEBUG } 22:24:04 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:04 DEBUG } 22:24:04 ERROR --- http response --- 22:24:04 DEBUG --- http response --- 22:24:04 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:24:04 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:24:04 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "instance": "10.142.0.107:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "instance": "10.142.0.43:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0.00030000000000001136" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-idrepo-2" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "ds-idrepo-0" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "2306.733333333333" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:24:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:24:05 DEBUG }, 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:24:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:24:05 DEBUG }, 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "6.5793678709827175" 22:24:05 DEBUG "2581708800" 22:24:05 DEBUG ] 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-idrepo-2" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "ds-idrepo-0" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "2283.0333333333333" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "instance": "10.142.0.107:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "instance": "10.142.0.43:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "4530.799999999999" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-idrepo-2" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "ds-idrepo-0" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-cts-1" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-idrepo-2" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "ds-idrepo-0" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "2303.4666666666667" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-idrepo-2" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "ds-idrepo-0" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-cts-1" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "11864263.089675961" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216245" 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "instance": "10.142.0.107:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "instance": "10.142.0.43:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-idrepo-2" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "ds-idrepo-0" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "0" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-cts-1" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "180261.03993971364" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "pod": "ds-cts-1" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "16.003122079879546" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216245" 22:24:05 INFO [http_cmd]: http status code OK 22:24:05 DEBUG --- status code --- 22:24:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:24:05 DEBUG --- http response --- 22:24:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:24:05 DEBUG ----- output ----- 22:24:05 DEBUG { 22:24:05 DEBUG "status": "success", 22:24:05 DEBUG "data": { 22:24:05 DEBUG "resultType": "vector", 22:24:05 DEBUG "result": [ 22:24:05 DEBUG { 22:24:05 DEBUG "metric": { 22:24:05 DEBUG "instance": "10.142.0.107:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG [...] 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG 22:24:05 DEBUG "instance": "10.142.0.43:9100" 22:24:05 DEBUG }, 22:24:05 DEBUG "value": [ 22:24:05 DEBUG 1713216245, 22:24:05 DEBUG "13868696576" 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG ] 22:24:05 DEBUG } 22:24:05 DEBUG } 22:24:05 INFO 22:24:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:24:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:24:05 INFO [loop_until]: OK (rc = 0) 22:24:05 DEBUG --- stdout --- 22:24:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12440m 8665Mi am-7d597cf7d6-8vr8f 11702m 9069Mi am-7d597cf7d6-9wz9h 15897m 9233Mi ds-cts-0 14928m 8472Mi ds-cts-1 11095m 8533Mi ds-cts-2 10828m 8719Mi ds-idrepo-0 13330m 4956Mi ds-idrepo-1 1987m 2532Mi ds-idrepo-2 2020m 2514Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6630m 2370Mi 22:24:05 DEBUG --- stderr --- 22:24:05 DEBUG 22:24:06 INFO 22:24:06 INFO [loop_until]: kubectl --namespace=xlou top node 22:24:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:24:06 INFO [loop_until]: OK (rc = 0) 22:24:06 DEBUG --- stdout --- 22:24:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11335m 37% 10773Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13134m 43% 10753Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17564m 58% 3063Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2035m 6% 5369Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15904m 53% 12771Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2961Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12602m 42% 10569Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2080m 6% 4770Mi 4% gke-xlou-cdm-ds-0f601751-j488 10534m 35% 10316Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2460Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17259m 57% 10885Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14015m 46% 7140Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6756m 42% 6216Mi 10% 22:24:06 DEBUG --- stderr --- 22:24:06 DEBUG 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "2306.8333333333335" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "status": "success", 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 ERROR [http_cmd]: ERROR 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "data": { 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG --- status code --- 22:25:05 ERROR --- status code --- 22:25:05 DEBUG "data": { 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG "result": [ 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG --- http response --- 22:25:05 ERROR --- http response --- 22:25:05 DEBUG "result": [ 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG { 22:25:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG "metric": { 22:25:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:25:05 DEBUG "pod": "ds-idrepo-2" 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG "instance": "10.142.0.107:9100" 22:25:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "data": { 22:25:05 DEBUG "data": { 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "result": [ 22:25:05 DEBUG "result": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG [...] 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG [...] 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-idrepo-2" 22:25:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:25:05 DEBUG "pod": "ds-cts-2" 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "instance": "10.142.0.43:9100" 22:25:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG [...] 22:25:05 DEBUG [...] 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG "0" 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "12416.666666666664" 22:25:05 DEBUG "0.08" 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG "pod": "ds-idrepo-0" 22:25:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG "413671.5" 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "2306.4" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "instance": "10.142.0.107:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "instance": "10.142.0.43:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0.00036666666666652263" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "6.711348732583964" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "ds-idrepo-2" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-idrepo-0" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "2583879680" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "2302.9666666666667" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "instance": "10.142.0.107:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "instance": "10.142.0.43:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "3321.0666666666666" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "ds-idrepo-2" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-idrepo-0" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "2308.266666666667" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "ds-idrepo-2" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-idrepo-0" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-cts-2" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "ds-idrepo-2" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-idrepo-0" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-cts-2" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "38685550.59147459" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "instance": "10.142.0.107:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "instance": "10.142.0.43:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0.0006666666666666673" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "ds-idrepo-2" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-idrepo-0" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "0" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:25:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "pod": "ds-cts-2" 22:25:05 DEBUG "pod": "ds-cts-2" 22:25:05 DEBUG }, 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "38000.30111394661" 22:25:05 DEBUG "7.3399415161619945" 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216305" 22:25:05 INFO [http_cmd]: http status code OK 22:25:05 DEBUG --- status code --- 22:25:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:25:05 DEBUG --- http response --- 22:25:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:25:05 DEBUG ----- output ----- 22:25:05 DEBUG { 22:25:05 DEBUG "status": "success", 22:25:05 DEBUG "data": { 22:25:05 DEBUG "resultType": "vector", 22:25:05 DEBUG "result": [ 22:25:05 DEBUG { 22:25:05 DEBUG "metric": { 22:25:05 DEBUG "instance": "10.142.0.107:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG [...] 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG 22:25:05 DEBUG "instance": "10.142.0.43:9100" 22:25:05 DEBUG }, 22:25:05 DEBUG "value": [ 22:25:05 DEBUG 1713216305, 22:25:05 DEBUG "13869483008" 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG ] 22:25:05 DEBUG } 22:25:05 DEBUG } 22:25:05 INFO 22:25:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:25:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:25:05 INFO [loop_until]: OK (rc = 0) 22:25:05 DEBUG --- stdout --- 22:25:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12680m 8663Mi am-7d597cf7d6-8vr8f 11747m 9068Mi am-7d597cf7d6-9wz9h 15685m 9237Mi ds-cts-0 15118m 8492Mi ds-cts-1 10804m 8549Mi ds-cts-2 10962m 8700Mi ds-idrepo-0 13376m 4955Mi ds-idrepo-1 1971m 2548Mi ds-idrepo-2 1939m 2597Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6479m 2369Mi 22:25:05 DEBUG --- stderr --- 22:25:05 DEBUG 22:25:06 INFO 22:25:06 INFO [loop_until]: kubectl --namespace=xlou top node 22:25:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:25:06 INFO [loop_until]: OK (rc = 0) 22:25:06 DEBUG --- stdout --- 22:25:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5066Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10429m 34% 10775Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13154m 44% 10754Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17618m 59% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2036m 6% 5433Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16520m 55% 12786Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2956Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12597m 42% 10568Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2093m 7% 4764Mi 4% gke-xlou-cdm-ds-0f601751-j488 11026m 36% 10316Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2440Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17373m 58% 10886Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14080m 47% 7133Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6788m 42% 6212Mi 10% 22:25:06 DEBUG --- stderr --- 22:25:06 DEBUG 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "2291.3" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-2" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "ds-cts-2" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:26:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:26:05 DEBUG }, 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG [...] 22:26:05 DEBUG [...] 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:26:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG }, 22:26:05 DEBUG }, 22:26:05 DEBUG { 22:26:05 DEBUG "value": [ 22:26:05 DEBUG "value": [ 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "data": { 22:26:05 DEBUG "414243.5" 22:26:05 DEBUG "0.08" 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG ] 22:26:05 DEBUG ] 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG "result": [ 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG { 22:26:05 DEBUG ] 22:26:05 DEBUG ] 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG "metric": { 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG "instance": "10.142.0.107:9100" 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 DEBUG }, 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG "value": [ 22:26:05 DEBUG ----- output ----- 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG 22:26:05 DEBUG { 22:26:05 DEBUG --- status code --- 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 DEBUG 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG 22:26:05 DEBUG "data": { 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [...] 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG "result": [ 22:26:05 DEBUG 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "instance": "10.142.0.43:9100" 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:26:05 DEBUG }, 22:26:05 DEBUG "data": { 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "result": [ 22:26:05 DEBUG 22:26:05 DEBUG "0.00036666666666652263" 22:26:05 DEBUG { 22:26:05 DEBUG 22:26:05 DEBUG ] 22:26:05 DEBUG "metric": { 22:26:05 DEBUG 22:26:05 DEBUG } 22:26:05 DEBUG "instance": "10.142.0.107:9100" 22:26:05 DEBUG [...] 22:26:05 DEBUG ] 22:26:05 DEBUG }, 22:26:05 DEBUG 22:26:05 DEBUG } 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG } 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:26:05 DEBUG 22:26:05 DEBUG }, 22:26:05 DEBUG [...] 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG 22:26:05 DEBUG "2291.8333333333335" 22:26:05 DEBUG 22:26:05 DEBUG ] 22:26:05 DEBUG "instance": "10.142.0.43:9100" 22:26:05 DEBUG } 22:26:05 DEBUG }, 22:26:05 DEBUG ] 22:26:05 DEBUG "value": [ 22:26:05 DEBUG } 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG } 22:26:05 DEBUG "11962.23333333333" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 ERROR [http_cmd]: ERROR 22:26:05 ERROR --- status code --- 22:26:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:26:05 ERROR --- http response --- 22:26:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:26:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-2" 22:26:05 DEBUG }, 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG "value": [ 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG 22:26:05 DEBUG { 22:26:05 DEBUG 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG 22:26:05 DEBUG "data": { 22:26:05 DEBUG [...] 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG 22:26:05 DEBUG "result": [ 22:26:05 DEBUG 22:26:05 DEBUG { 22:26:05 DEBUG 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-0" 22:26:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:26:05 DEBUG }, 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG 22:26:05 DEBUG "0" 22:26:05 DEBUG 22:26:05 DEBUG ] 22:26:05 DEBUG 22:26:05 DEBUG } 22:26:05 DEBUG [...] 22:26:05 DEBUG ] 22:26:05 DEBUG 22:26:05 DEBUG } 22:26:05 DEBUG 22:26:05 DEBUG } 22:26:05 DEBUG 22:26:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "2297.9" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-2" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "ds-idrepo-0" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "6.587101491820496" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "2585337856" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "instance": "10.142.0.107:9100" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "instance": "10.142.0.43:9100" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "4331" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-2" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "ds-idrepo-0" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "2288.3" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-2" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "ds-idrepo-0" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-2" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "ds-idrepo-0" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-cts-1" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-cts-1" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-idrepo-2" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "ds-idrepo-0" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "instance": "10.142.0.107:9100" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "instance": "10.142.0.43:9100" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0.000333333333333341" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-cts-1" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "pod": "ds-cts-1" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "0.000023157764206206687" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216365" 22:26:05 INFO [http_cmd]: http status code OK 22:26:05 DEBUG --- status code --- 22:26:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:26:05 DEBUG --- http response --- 22:26:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:26:05 DEBUG ----- output ----- 22:26:05 DEBUG { 22:26:05 DEBUG "status": "success", 22:26:05 DEBUG "data": { 22:26:05 DEBUG "resultType": "vector", 22:26:05 DEBUG "result": [ 22:26:05 DEBUG { 22:26:05 DEBUG "metric": { 22:26:05 DEBUG "instance": "10.142.0.107:9100" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG [...] 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG 22:26:05 DEBUG "instance": "10.142.0.43:9100" 22:26:05 DEBUG }, 22:26:05 DEBUG "value": [ 22:26:05 DEBUG 1713216365, 22:26:05 DEBUG "13869943808" 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG ] 22:26:05 DEBUG } 22:26:05 DEBUG } 22:26:05 INFO 22:26:05 INFO [loop_until]: kubectl --namespace=xlou top pods 22:26:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:26:06 INFO [loop_until]: OK (rc = 0) 22:26:06 DEBUG --- stdout --- 22:26:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12815m 8666Mi am-7d597cf7d6-8vr8f 12084m 9068Mi am-7d597cf7d6-9wz9h 16308m 9234Mi ds-cts-0 15633m 8479Mi ds-cts-1 11300m 8539Mi ds-cts-2 10703m 8713Mi ds-idrepo-0 13619m 4955Mi ds-idrepo-1 1975m 2565Mi ds-idrepo-2 1979m 2597Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6567m 2371Mi 22:26:06 DEBUG --- stderr --- 22:26:06 DEBUG 22:26:06 INFO 22:26:06 INFO [loop_until]: kubectl --namespace=xlou top node 22:26:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:26:06 INFO [loop_until]: OK (rc = 0) 22:26:06 DEBUG --- stdout --- 22:26:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5074Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11275m 37% 10782Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13354m 44% 10756Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17062m 57% 3059Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2053m 6% 5454Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16787m 56% 12789Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2963Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12696m 42% 10562Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2077m 6% 4793Mi 4% gke-xlou-cdm-ds-0f601751-j488 10892m 36% 10319Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2452Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17223m 57% 10896Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14062m 47% 7123Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6757m 42% 6211Mi 10% 22:26:06 DEBUG --- stderr --- 22:26:06 DEBUG 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "2287.2666666666664" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "2307.6" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:27:05 ERROR [http_cmd]: ERROR 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 ERROR --- status code --- 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:27:05 DEBUG { 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 ERROR --- http response --- 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "data": { 22:27:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:27:05 DEBUG "data": { 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "resultType": "vector", 22:27:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG "result": [ 22:27:05 DEBUG "result": [ 22:27:05 DEBUG "result": [ 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG { 22:27:05 DEBUG { 22:27:05 DEBUG { 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-idrepo-2" 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:27:05 DEBUG "instance": "10.142.0.107:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG }, 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG "value": [ 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG "value": [ 22:27:05 DEBUG { 22:27:05 DEBUG 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG 22:27:05 DEBUG "data": { 22:27:05 DEBUG [...] 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG [...] 22:27:05 DEBUG [...] 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG "result": [ 22:27:05 DEBUG 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG 22:27:05 DEBUG { 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-cts-2" 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "instance": "10.142.0.107:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG "data": { 22:27:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:27:05 DEBUG "instance": "10.142.0.43:9100" 22:27:05 DEBUG "data": { 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG }, 22:27:05 DEBUG }, 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "result": [ 22:27:05 DEBUG "value": [ 22:27:05 DEBUG "value": [ 22:27:05 DEBUG "result": [ 22:27:05 DEBUG 22:27:05 DEBUG "0" 22:27:05 DEBUG { 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG { 22:27:05 DEBUG 22:27:05 DEBUG ] 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "0.08" 22:27:05 DEBUG "12092.899999999998" 22:27:05 DEBUG "metric": { 22:27:05 DEBUG 22:27:05 DEBUG } 22:27:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:27:05 DEBUG ] 22:27:05 DEBUG ] 22:27:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:27:05 DEBUG [...] 22:27:05 DEBUG ] 22:27:05 DEBUG }, 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 DEBUG }, 22:27:05 DEBUG 22:27:05 DEBUG } 22:27:05 DEBUG "value": [ 22:27:05 DEBUG ] 22:27:05 DEBUG ] 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG } 22:27:05 DEBUG 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 DEBUG 22:27:05 DEBUG "instance": "10.142.0.43:9100" 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG }, 22:27:05 DEBUG [...] 22:27:05 DEBUG [...] 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "0.00030000000000001136" 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG ] 22:27:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:27:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:27:05 DEBUG } 22:27:05 DEBUG }, 22:27:05 DEBUG }, 22:27:05 DEBUG ] 22:27:05 DEBUG "value": [ 22:27:05 DEBUG "value": [ 22:27:05 DEBUG } 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG } 22:27:05 DEBUG "413734" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG "2286.7" 22:27:05 DEBUG ] 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-idrepo-2" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "ds-idrepo-0" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-idrepo-2" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "ds-idrepo-0" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "6.588027699749605" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "2585034752" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "instance": "10.142.0.107:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "instance": "10.142.0.43:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "3352.1666666666665" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-idrepo-2" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "ds-idrepo-0" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-idrepo-2" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "ds-idrepo-0" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "2280.9" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-idrepo-2" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "ds-idrepo-0" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "ds-idrepo-2" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "ds-idrepo-0" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "instance": "10.142.0.107:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "instance": "10.142.0.43:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0.000333333333333341" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0.00001915901037910819" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "0" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216425" 22:27:05 INFO [http_cmd]: http status code OK 22:27:05 DEBUG --- status code --- 22:27:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:27:05 DEBUG --- http response --- 22:27:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:27:05 DEBUG ----- output ----- 22:27:05 DEBUG { 22:27:05 DEBUG "status": "success", 22:27:05 DEBUG "data": { 22:27:05 DEBUG "resultType": "vector", 22:27:05 DEBUG "result": [ 22:27:05 DEBUG { 22:27:05 DEBUG "metric": { 22:27:05 DEBUG "instance": "10.142.0.107:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG [...] 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG 22:27:05 DEBUG "instance": "10.142.0.43:9100" 22:27:05 DEBUG }, 22:27:05 DEBUG "value": [ 22:27:05 DEBUG 1713216425, 22:27:05 DEBUG "13870646272" 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG ] 22:27:05 DEBUG } 22:27:05 DEBUG } 22:27:06 INFO 22:27:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:27:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:27:06 INFO [loop_until]: OK (rc = 0) 22:27:06 DEBUG --- stdout --- 22:27:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12389m 8669Mi am-7d597cf7d6-8vr8f 11648m 9068Mi am-7d597cf7d6-9wz9h 16579m 9239Mi ds-cts-0 15316m 8484Mi ds-cts-1 11174m 8541Mi ds-cts-2 10656m 8705Mi ds-idrepo-0 13216m 4955Mi ds-idrepo-1 2019m 2564Mi ds-idrepo-2 1968m 2597Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6799m 2373Mi 22:27:06 DEBUG --- stderr --- 22:27:06 DEBUG 22:27:06 INFO 22:27:06 INFO [loop_until]: kubectl --namespace=xlou top node 22:27:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:27:06 INFO [loop_until]: OK (rc = 0) 22:27:06 DEBUG --- stdout --- 22:27:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5073Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11090m 37% 10778Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13077m 43% 10742Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17504m 58% 3066Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2012m 6% 5425Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16451m 55% 12819Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2944Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12565m 42% 10567Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2092m 7% 4796Mi 4% gke-xlou-cdm-ds-0f601751-j488 10836m 36% 10313Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2442Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17220m 57% 10894Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14007m 46% 7137Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6701m 42% 6213Mi 10% 22:27:06 DEBUG --- stderr --- 22:27:06 DEBUG 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- http response --- 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG { 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG { 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "data": { 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 DEBUG "status": "success", 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "data": { 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG { 22:28:05 DEBUG { 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "result": [ 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG { 22:28:05 DEBUG "result": [ 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG "data": { 22:28:05 DEBUG "data": { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:28:05 DEBUG "metric": { 22:28:05 DEBUG { 22:28:05 ERROR [http_cmd]: ERROR 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:28:05 DEBUG }, 22:28:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:28:05 DEBUG "metric": { 22:28:05 ERROR --- status code --- 22:28:05 DEBUG "result": [ 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG "result": [ 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG }, 22:28:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:28:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:28:05 DEBUG { 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG { 22:28:05 DEBUG { 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG "value": [ 22:28:05 DEBUG }, 22:28:05 ERROR --- http response --- 22:28:05 DEBUG "metric": { 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "metric": { 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:28:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG "data": { 22:28:05 DEBUG "instance": "10.142.0.107:9100" 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG }, 22:28:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG }, 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "result": [ 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG "value": [ 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG { 22:28:05 DEBUG { 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "ds-idrepo-2" 22:28:05 DEBUG "data": { 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG }, 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG [...] 22:28:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:28:05 DEBUG }, 22:28:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "result": [ 22:28:05 DEBUG 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG }, 22:28:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG { 22:28:05 DEBUG 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG }, 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "metric": { 22:28:05 DEBUG 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:28:05 DEBUG 22:28:05 DEBUG "instance": "10.142.0.107:9100" 22:28:05 DEBUG "instance": "10.142.0.43:9100" 22:28:05 DEBUG "2299.4" 22:28:05 DEBUG ] 22:28:05 DEBUG "2299.5666666666666" 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG }, 22:28:05 DEBUG [...] 22:28:05 DEBUG }, 22:28:05 DEBUG }, 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG "413333.5" 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "value": [ 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG "2303.9" 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "0.0003666666666669964" 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG ] 22:28:05 DEBUG "pod": "ds-cts-2" 22:28:05 DEBUG 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG }, 22:28:05 DEBUG [...] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG "0" 22:28:05 DEBUG 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG "instance": "10.142.0.43:9100" 22:28:05 DEBUG } 22:28:05 DEBUG }, 22:28:05 DEBUG ] 22:28:05 DEBUG "value": [ 22:28:05 DEBUG } 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG } 22:28:05 DEBUG "12290.33333333333" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:28:05 DEBUG "pod": "ds-idrepo-2" 22:28:05 DEBUG }, 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:28:05 DEBUG "pod": "ds-idrepo-0" 22:28:05 DEBUG }, 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0.08" 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "ds-idrepo-2" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "ds-idrepo-0" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "2584735744" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "6.5958530893043985" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "instance": "10.142.0.107:9100" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "instance": "10.142.0.43:9100" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "3023.366666666667" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "ds-idrepo-2" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "ds-idrepo-0" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "ds-idrepo-2" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "ds-idrepo-0" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "2307.1" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "ds-idrepo-2" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "ds-idrepo-0" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "4400.673280091681" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "ds-idrepo-2" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "ds-idrepo-0" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:28:05 DEBUG }, 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:28:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:28:05 DEBUG }, 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0.0011488020628156097" 22:28:05 DEBUG "2397633.4921032838" 22:28:05 DEBUG ] 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "instance": "10.142.0.107:9100" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "instance": "10.142.0.43:9100" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "0" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216485" 22:28:05 INFO [http_cmd]: http status code OK 22:28:05 DEBUG --- status code --- 22:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:28:05 DEBUG --- http response --- 22:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:28:05 DEBUG ----- output ----- 22:28:05 DEBUG { 22:28:05 DEBUG "status": "success", 22:28:05 DEBUG "data": { 22:28:05 DEBUG "resultType": "vector", 22:28:05 DEBUG "result": [ 22:28:05 DEBUG { 22:28:05 DEBUG "metric": { 22:28:05 DEBUG "instance": "10.142.0.107:9100" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG [...] 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG 22:28:05 DEBUG "instance": "10.142.0.43:9100" 22:28:05 DEBUG }, 22:28:05 DEBUG "value": [ 22:28:05 DEBUG 1713216485, 22:28:05 DEBUG "13871324160" 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG ] 22:28:05 DEBUG } 22:28:05 DEBUG } 22:28:06 INFO 22:28:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:28:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:28:06 INFO [loop_until]: OK (rc = 0) 22:28:06 DEBUG --- stdout --- 22:28:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13063m 8667Mi am-7d597cf7d6-8vr8f 11662m 9069Mi am-7d597cf7d6-9wz9h 16149m 9237Mi ds-cts-0 15737m 8484Mi ds-cts-1 11031m 8544Mi ds-cts-2 10752m 8719Mi ds-idrepo-0 13577m 4955Mi ds-idrepo-1 1998m 2575Mi ds-idrepo-2 1893m 2598Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6616m 2375Mi 22:28:06 DEBUG --- stderr --- 22:28:06 DEBUG 22:28:06 INFO 22:28:06 INFO [loop_until]: kubectl --namespace=xlou top node 22:28:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:28:06 INFO [loop_until]: OK (rc = 0) 22:28:06 DEBUG --- stdout --- 22:28:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5068Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11096m 37% 10788Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13210m 44% 10751Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17692m 59% 3055Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1987m 6% 5439Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16589m 55% 12788Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2943Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12691m 42% 10565Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2092m 7% 4787Mi 4% gke-xlou-cdm-ds-0f601751-j488 10803m 36% 10336Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2444Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17340m 58% 10885Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13635m 45% 7120Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6787m 42% 6212Mi 10% 22:28:06 DEBUG --- stderr --- 22:28:06 DEBUG 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 ERROR [http_cmd]: ERROR 22:29:05 DEBUG --- status code --- 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 ERROR --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- status code --- 22:29:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 ERROR --- http response --- 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:29:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:29:05 DEBUG { 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG { 22:29:05 DEBUG "data": { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "result": [ 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG { 22:29:05 DEBUG "result": [ 22:29:05 DEBUG "metric": { 22:29:05 DEBUG { 22:29:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:29:05 DEBUG "metric": { 22:29:05 DEBUG }, 22:29:05 DEBUG "pod": "ds-idrepo-2" 22:29:05 DEBUG "value": [ 22:29:05 DEBUG }, 22:29:05 DEBUG 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:29:05 DEBUG 22:29:05 DEBUG }, 22:29:05 DEBUG "pod": "ds-cts-2" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 DEBUG "value": [ 22:29:05 DEBUG }, 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG "0" 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG ] 22:29:05 DEBUG "0" 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG { 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG "data": { 22:29:05 DEBUG { 22:29:05 DEBUG } 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG "result": [ 22:29:05 DEBUG "data": { 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "metric": { 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:29:05 DEBUG "result": [ 22:29:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG }, 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "value": [ 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:29:05 DEBUG 22:29:05 DEBUG "data": { 22:29:05 DEBUG }, 22:29:05 DEBUG 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG "result": [ 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG { 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "metric": { 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "ds-idrepo-2" 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG }, 22:29:05 DEBUG 22:29:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG }, 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG 22:29:05 DEBUG }, 22:29:05 DEBUG "0.08" 22:29:05 DEBUG [...] 22:29:05 DEBUG "value": [ 22:29:05 DEBUG ] 22:29:05 DEBUG 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG } 22:29:05 DEBUG 22:29:05 DEBUG "2294.6666666666665" 22:29:05 DEBUG ] 22:29:05 DEBUG 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG "pod": "ds-idrepo-0" 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 DEBUG }, 22:29:05 DEBUG ] 22:29:05 DEBUG "value": [ 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG } 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG "0" 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG ] 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG } 22:29:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "2293.7" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "instance": "10.142.0.107:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "instance": "10.142.0.43:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "12978.73333333333" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "2584535040" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "413197" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "2293.0666666666666" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "instance": "10.142.0.107:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "instance": "10.142.0.43:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0.00043333333333303397" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "ds-idrepo-2" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "ds-idrepo-0" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "6.573591163045106" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "instance": "10.142.0.107:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "instance": "10.142.0.43:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "4987.733333333333" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "ds-idrepo-2" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "ds-idrepo-0" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "ds-idrepo-2" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "ds-idrepo-0" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "2298.6" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "ds-idrepo-2" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "ds-idrepo-0" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "instance": "10.142.0.107:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "instance": "10.142.0.43:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0.00033333333333333365" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:29:05 DEBUG "pod": "ds-idrepo-2" 22:29:05 DEBUG }, 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:29:05 DEBUG "pod": "ds-idrepo-0" 22:29:05 DEBUG }, 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0.000012876701359064526" 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "0" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216545" 22:29:05 INFO [http_cmd]: http status code OK 22:29:05 DEBUG --- status code --- 22:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:29:05 DEBUG --- http response --- 22:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:29:05 DEBUG ----- output ----- 22:29:05 DEBUG { 22:29:05 DEBUG "status": "success", 22:29:05 DEBUG "data": { 22:29:05 DEBUG "resultType": "vector", 22:29:05 DEBUG "result": [ 22:29:05 DEBUG { 22:29:05 DEBUG "metric": { 22:29:05 DEBUG "instance": "10.142.0.107:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG [...] 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG 22:29:05 DEBUG "instance": "10.142.0.43:9100" 22:29:05 DEBUG }, 22:29:05 DEBUG "value": [ 22:29:05 DEBUG 1713216545, 22:29:05 DEBUG "13872114688" 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG ] 22:29:05 DEBUG } 22:29:05 DEBUG } 22:29:06 INFO 22:29:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:29:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:29:06 INFO [loop_until]: OK (rc = 0) 22:29:06 DEBUG --- stdout --- 22:29:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13112m 8670Mi am-7d597cf7d6-8vr8f 11869m 9067Mi am-7d597cf7d6-9wz9h 15638m 9236Mi ds-cts-0 15451m 8488Mi ds-cts-1 10954m 8548Mi ds-cts-2 10643m 8710Mi ds-idrepo-0 13867m 4956Mi ds-idrepo-1 1995m 2576Mi ds-idrepo-2 1998m 2598Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6704m 2377Mi 22:29:06 DEBUG --- stderr --- 22:29:06 DEBUG 22:29:07 INFO 22:29:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:29:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:29:07 INFO [loop_until]: OK (rc = 0) 22:29:07 DEBUG --- stdout --- 22:29:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5073Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11132m 37% 10795Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12827m 42% 10764Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17672m 59% 3068Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2062m 6% 5436Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16585m 55% 12794Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2952Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12672m 42% 10555Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2088m 6% 4796Mi 4% gke-xlou-cdm-ds-0f601751-j488 10717m 35% 10337Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2459Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17300m 57% 10885Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14063m 47% 7133Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6762m 42% 6211Mi 10% 22:29:07 DEBUG --- stderr --- 22:29:07 DEBUG 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "2285.6" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "ds-idrepo-2" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "ds-idrepo-0" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0.08" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "2285.6666666666665" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 ERROR [http_cmd]: ERROR 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:05 ERROR --- status code --- 22:30:05 DEBUG ----- output ----- 22:30:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:30:05 DEBUG { 22:30:05 ERROR --- http response --- 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:30:05 DEBUG "metric": { 22:30:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:30:05 DEBUG "instance": "10.142.0.107:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "instance": "10.142.0.43:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0.00030000000000001136" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:05 DEBUG "result": [ 22:30:05 DEBUG "result": [ 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG { 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "pod": "ds-idrepo-2" 22:30:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:30:05 DEBUG "data": { 22:30:05 DEBUG }, 22:30:05 DEBUG }, 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "value": [ 22:30:05 DEBUG "value": [ 22:30:05 DEBUG "result": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG { 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "metric": { 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:30:05 DEBUG [...] 22:30:05 DEBUG [...] 22:30:05 DEBUG }, 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "ds-cts-2" 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG 22:30:05 DEBUG }, 22:30:05 DEBUG }, 22:30:05 DEBUG [...] 22:30:05 DEBUG "value": [ 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG 22:30:05 DEBUG "0" 22:30:05 DEBUG "0" 22:30:05 DEBUG 22:30:05 DEBUG ] 22:30:05 DEBUG ] 22:30:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 DEBUG }, 22:30:05 DEBUG ] 22:30:05 DEBUG ] 22:30:05 DEBUG "value": [ 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 DEBUG "2587234304" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "instance": "10.142.0.107:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "instance": "10.142.0.43:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "11927.466666666665" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "413191" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "2287.2666666666664" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "ds-idrepo-2" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "ds-idrepo-0" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "6.636637633074284" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "2282.5" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "instance": "10.142.0.107:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "instance": "10.142.0.43:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "3021" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "ds-idrepo-2" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "ds-idrepo-0" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "ds-idrepo-2" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "ds-idrepo-0" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "ds-idrepo-2" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "ds-idrepo-0" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "3464.6699635866494" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "instance": "10.142.0.107:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "instance": "10.142.0.43:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0.0003333333333333262" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG { 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "ds-idrepo-2" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "ds-idrepo-0" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "0" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216605" 22:30:05 INFO [http_cmd]: http status code OK 22:30:05 DEBUG --- status code --- 22:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:30:05 DEBUG --- http response --- 22:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:30:05 DEBUG ----- output ----- 22:30:05 DEBUG { 22:30:05 DEBUG "status": "success", 22:30:05 DEBUG "data": { 22:30:05 DEBUG "resultType": "vector", 22:30:05 DEBUG "result": [ 22:30:05 DEBUG { 22:30:05 DEBUG "metric": { 22:30:05 DEBUG "instance": "10.142.0.107:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG [...] 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG 22:30:05 DEBUG "instance": "10.142.0.43:9100" 22:30:05 DEBUG }, 22:30:05 DEBUG "value": [ 22:30:05 DEBUG 1713216605, 22:30:05 DEBUG "13872845824" 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG ] 22:30:05 DEBUG } 22:30:05 DEBUG } 22:30:06 INFO 22:30:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:30:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:30:06 INFO [loop_until]: OK (rc = 0) 22:30:06 DEBUG --- stdout --- 22:30:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12452m 8667Mi am-7d597cf7d6-8vr8f 11891m 9077Mi am-7d597cf7d6-9wz9h 16017m 9237Mi ds-cts-0 15054m 8490Mi ds-cts-1 10966m 8554Mi ds-cts-2 11139m 8724Mi ds-idrepo-0 13681m 4956Mi ds-idrepo-1 2040m 2608Mi ds-idrepo-2 1938m 2599Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6703m 2378Mi 22:30:06 DEBUG --- stderr --- 22:30:06 DEBUG 22:30:07 INFO 22:30:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:30:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:30:07 INFO [loop_until]: OK (rc = 0) 22:30:07 DEBUG --- stdout --- 22:30:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5078Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11078m 37% 10799Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13178m 44% 10750Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16933m 56% 3063Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2067m 6% 5449Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16328m 54% 12813Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2958Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12602m 42% 10572Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2043m 6% 4842Mi 4% gke-xlou-cdm-ds-0f601751-j488 10846m 36% 10347Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17251m 57% 10884Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13975m 46% 7141Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6809m 42% 6211Mi 10% 22:30:07 DEBUG --- stderr --- 22:30:07 DEBUG 22:30:22 WARNING File /tmp/lodemon_data-20240415_215803/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 22:30:22] "GET /monitoring/average?start_time=20240415_210107&stop_time=20240415_212821 HTTP/1.1" 200 - 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "ds-idrepo-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-idrepo-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216665" 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0.08" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG { 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG "data": { 22:31:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG }, 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "value": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "instance": "10.142.0.107:9100" 22:31:05 DEBUG 22:31:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:31:05 DEBUG }, 22:31:05 DEBUG 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG [...] 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG }, 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG 22:31:05 DEBUG "instance": "10.142.0.43:9100" 22:31:05 DEBUG "0" 22:31:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:31:05 DEBUG }, 22:31:05 DEBUG ] 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG } 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG ] 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0.00026666666666651885" 22:31:05 DEBUG } 22:31:05 DEBUG "58.833333333333336" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "2091745280" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "58.96666666666667" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216665" 22:31:05 ERROR [http_cmd]: ERROR 22:31:05 ERROR --- status code --- 22:31:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:31:05 ERROR --- http response --- 22:31:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:31:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "ds-idrepo-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-cts-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "instance": "10.142.0.107:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "instance": "10.142.0.43:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "12908.399999999998" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "331715.5" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "2298.233333333333" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "ds-idrepo-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-idrepo-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0.4392940259869643" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "59" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "182419.21250155158" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "instance": "10.142.0.107:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "instance": "10.142.0.43:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "4735.799999999999" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "ds-idrepo-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-idrepo-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "ds-idrepo-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-idrepo-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "ds-idrepo-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-idrepo-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "instance": "10.142.0.107:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "instance": "10.142.0.43:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0.0006666666666666598" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-cts-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "ds-idrepo-2" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-idrepo-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "pod": "ds-cts-0" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "0" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216665" 22:31:05 INFO [http_cmd]: http status code OK 22:31:05 DEBUG --- status code --- 22:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:31:05 DEBUG --- http response --- 22:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:31:05 DEBUG ----- output ----- 22:31:05 DEBUG { 22:31:05 DEBUG "status": "success", 22:31:05 DEBUG "data": { 22:31:05 DEBUG "resultType": "vector", 22:31:05 DEBUG "result": [ 22:31:05 DEBUG { 22:31:05 DEBUG "metric": { 22:31:05 DEBUG "instance": "10.142.0.107:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG [...] 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG 22:31:05 DEBUG "instance": "10.142.0.43:9100" 22:31:05 DEBUG }, 22:31:05 DEBUG "value": [ 22:31:05 DEBUG 1713216665, 22:31:05 DEBUG "13873374208" 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG ] 22:31:05 DEBUG } 22:31:05 DEBUG } 22:31:06 INFO 22:31:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:31:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:31:06 INFO [loop_until]: OK (rc = 0) 22:31:06 DEBUG --- stdout --- 22:31:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2189m 8665Mi am-7d597cf7d6-8vr8f 2376m 7996Mi am-7d597cf7d6-9wz9h 3499m 9233Mi ds-cts-0 7489m 8457Mi ds-cts-1 5210m 8559Mi ds-cts-2 5178m 8688Mi ds-idrepo-0 1325m 4956Mi ds-idrepo-1 254m 2609Mi ds-idrepo-2 195m 2598Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 2256m 2109Mi 22:31:06 DEBUG --- stderr --- 22:31:06 DEBUG 22:31:07 INFO 22:31:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:31:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:31:07 INFO [loop_until]: OK (rc = 0) 22:31:07 DEBUG --- stdout --- 22:31:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5074Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5702m 19% 10800Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2921m 9% 10781Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 916m 3% 3039Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 334m 1% 5451Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 8436m 28% 12797Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2962Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2601m 8% 9517Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 488m 1% 4826Mi 4% gke-xlou-cdm-ds-0f601751-j488 5825m 19% 10332Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2457Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 3216m 10% 10936Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 2735m 9% 7137Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 2592m 16% 5941Mi 10% 22:31:07 DEBUG --- stderr --- 22:31:07 DEBUG 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:32:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:32:05 DEBUG }, 22:32:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:32:05 DEBUG }, 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG }, 22:32:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:32:05 DEBUG }, 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG }, 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 DEBUG "value": [ 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "2496.0666666666666" 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG "2765135872" 22:32:05 DEBUG ] 22:32:05 DEBUG "0.08" 22:32:05 DEBUG ] 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-idrepo-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "ds-idrepo-0" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "instance": "10.142.0.107:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "instance": "10.142.0.43:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0.00043333333333350765" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "instance": "10.142.0.107:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "instance": "10.142.0.43:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "12367.599999999997" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "2495.866666666667" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "2395.4666666666667" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 ERROR [http_cmd]: ERROR 22:32:05 ERROR --- status code --- 22:32:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:32:05 ERROR --- http response --- 22:32:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:32:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-idrepo-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "ds-cts-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "6.5854830337420465" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "281271.5" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-idrepo-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "ds-idrepo-0" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "2496.0333333333333" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-cts-1" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-cts-1" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "instance": "10.142.0.107:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "instance": "10.142.0.43:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "3323.4333333333334" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-idrepo-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "ds-idrepo-0" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-idrepo-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "ds-idrepo-0" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:05 DEBUG } 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG ] 22:32:05 DEBUG { 22:32:05 DEBUG } 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG } 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-idrepo-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "ds-idrepo-0" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-cts-1" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "instance": "10.142.0.107:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "instance": "10.142.0.43:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-idrepo-2" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "ds-idrepo-0" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "pod": "ds-cts-1" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "0" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216725" 22:32:05 INFO [http_cmd]: http status code OK 22:32:05 DEBUG --- status code --- 22:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:05 DEBUG --- http response --- 22:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:05 DEBUG ----- output ----- 22:32:05 DEBUG { 22:32:05 DEBUG "status": "success", 22:32:05 DEBUG "data": { 22:32:05 DEBUG "resultType": "vector", 22:32:05 DEBUG "result": [ 22:32:05 DEBUG { 22:32:05 DEBUG "metric": { 22:32:05 DEBUG "instance": "10.142.0.107:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG [...] 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG 22:32:05 DEBUG "instance": "10.142.0.43:9100" 22:32:05 DEBUG }, 22:32:05 DEBUG "value": [ 22:32:05 DEBUG 1713216725, 22:32:05 DEBUG "13873992704" 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG ] 22:32:05 DEBUG } 22:32:05 DEBUG } 22:32:06 INFO 22:32:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:32:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:06 INFO [loop_until]: OK (rc = 0) 22:32:06 DEBUG --- stdout --- 22:32:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13639m 8666Mi am-7d597cf7d6-8vr8f 12581m 7990Mi am-7d597cf7d6-9wz9h 16608m 9266Mi ds-cts-0 15480m 8471Mi ds-cts-1 11272m 8550Mi ds-cts-2 10845m 8714Mi ds-idrepo-0 14641m 4955Mi ds-idrepo-1 2094m 2608Mi ds-idrepo-2 2060m 2599Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7545m 3072Mi 22:32:06 DEBUG --- stderr --- 22:32:06 DEBUG 22:32:07 INFO 22:32:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:32:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:07 INFO [loop_until]: OK (rc = 0) 22:32:07 DEBUG --- stdout --- 22:32:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5073Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11251m 37% 10812Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14167m 47% 10748Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 20458m 68% 3289Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2161m 7% 5448Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16673m 55% 12813Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2963Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 13595m 45% 9490Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 2245m 7% 4833Mi 4% gke-xlou-cdm-ds-0f601751-j488 10999m 36% 10332Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2455Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 18458m 61% 10954Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14746m 49% 7131Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 7368m 46% 6888Mi 11% 22:32:07 DEBUG --- stderr --- 22:32:07 DEBUG 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG --- status code --- 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "instance": "10.142.0.107:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "instance": "10.142.0.107:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG 22:33:05 DEBUG "instance": "10.142.0.43:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "instance": "10.142.0.43:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0.08" 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0.00026666666666651885" 22:33:05 DEBUG ] 22:33:05 DEBUG "12242.666666666664" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 ERROR [http_cmd]: ERROR 22:33:05 ERROR --- status code --- 22:33:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:33:05 ERROR --- http response --- 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:33:05 DEBUG --- status code --- 22:33:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "1808.1333333333332" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG "result": [ 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG { 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG "metric": { 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG "pod": "ds-idrepo-2" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG [...] 22:33:05 DEBUG { 22:33:05 DEBUG 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG 22:33:05 DEBUG "data": { 22:33:05 DEBUG 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "pod": "ds-idrepo-0" 22:33:05 DEBUG "result": [ 22:33:05 DEBUG }, 22:33:05 DEBUG { 22:33:05 DEBUG "value": [ 22:33:05 DEBUG "metric": { 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:33:05 DEBUG "0" 22:33:05 DEBUG }, 22:33:05 DEBUG ] 22:33:05 DEBUG "value": [ 22:33:05 DEBUG } 22:33:05 DEBUG 22:33:05 DEBUG ] 22:33:05 DEBUG 22:33:05 DEBUG } 22:33:05 DEBUG 22:33:05 DEBUG } 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "3404087296" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "2077.233333333333" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "7.62506319006392" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "1810.2" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-idrepo-2" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "ds-cts-2" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "281849" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-idrepo-2" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "ds-idrepo-0" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "1787.7666666666667" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-cts-1" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "12029005.751211632" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "instance": "10.142.0.107:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "instance": "10.142.0.43:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "3230.6" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-cts-1" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-idrepo-2" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "ds-idrepo-0" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:05 DEBUG { 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG { 22:33:05 DEBUG "data": { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "result": [ 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG { 22:33:05 DEBUG "result": [ 22:33:05 DEBUG "metric": { 22:33:05 DEBUG { 22:33:05 DEBUG "pod": "ds-idrepo-2" 22:33:05 DEBUG "metric": { 22:33:05 DEBUG }, 22:33:05 DEBUG "pod": "ds-idrepo-2" 22:33:05 DEBUG "value": [ 22:33:05 DEBUG }, 22:33:05 DEBUG 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "ds-idrepo-0" 22:33:05 DEBUG 22:33:05 DEBUG }, 22:33:05 DEBUG "pod": "ds-idrepo-0" 22:33:05 DEBUG "value": [ 22:33:05 DEBUG }, 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG "0" 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG ] 22:33:05 DEBUG "0" 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-cts-1" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "314142157.5444265" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "instance": "10.142.0.107:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "instance": "10.142.0.43:9100" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-idrepo-2" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "ds-idrepo-0" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "0" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:05 DEBUG ----- output ----- 22:33:05 DEBUG { 22:33:05 DEBUG "status": "success", 22:33:05 DEBUG "data": { 22:33:05 DEBUG "resultType": "vector", 22:33:05 DEBUG "result": [ 22:33:05 DEBUG { 22:33:05 DEBUG "metric": { 22:33:05 DEBUG "pod": "ds-cts-1" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG [...] 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG 22:33:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:33:05 DEBUG }, 22:33:05 DEBUG "value": [ 22:33:05 DEBUG 1713216785, 22:33:05 DEBUG "17.118245331177064" 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG ] 22:33:05 DEBUG } 22:33:05 DEBUG } 22:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216785" 22:33:05 INFO [http_cmd]: http status code OK 22:33:05 DEBUG --- status code --- 22:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:05 DEBUG --- http response --- 22:33:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:06 DEBUG ----- output ----- 22:33:06 DEBUG { 22:33:06 DEBUG "status": "success", 22:33:06 DEBUG "data": { 22:33:06 DEBUG "resultType": "vector", 22:33:06 DEBUG "result": [ 22:33:06 DEBUG { 22:33:06 DEBUG "metric": { 22:33:06 DEBUG "instance": "10.142.0.107:9100" 22:33:06 DEBUG }, 22:33:06 DEBUG "value": [ 22:33:06 DEBUG 22:33:06 DEBUG 22:33:06 DEBUG 22:33:06 DEBUG [...] 22:33:06 DEBUG 22:33:06 DEBUG 22:33:06 DEBUG 22:33:06 DEBUG "instance": "10.142.0.43:9100" 22:33:06 DEBUG }, 22:33:06 DEBUG "value": [ 22:33:06 DEBUG 1713216785, 22:33:06 DEBUG "13874678784" 22:33:06 DEBUG ] 22:33:06 DEBUG } 22:33:06 DEBUG ] 22:33:06 DEBUG } 22:33:06 DEBUG } 22:33:06 INFO 22:33:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:33:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:33:06 INFO [loop_until]: OK (rc = 0) 22:33:06 DEBUG --- stdout --- 22:33:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 10007m 8669Mi am-7d597cf7d6-8vr8f 8698m 7992Mi am-7d597cf7d6-9wz9h 15468m 9269Mi ds-cts-0 13176m 8482Mi ds-cts-1 9752m 8578Mi ds-cts-2 9341m 8720Mi ds-idrepo-0 12071m 4956Mi ds-idrepo-1 1635m 2609Mi ds-idrepo-2 1544m 2599Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 4m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7933m 4464Mi 22:33:06 DEBUG --- stderr --- 22:33:06 DEBUG 22:33:07 INFO 22:33:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:33:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:33:07 INFO [loop_until]: OK (rc = 0) 22:33:07 DEBUG --- stdout --- 22:33:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5082Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9130m 30% 10818Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10132m 33% 10752Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 15576m 52% 3632Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1549m 5% 5447Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 13964m 46% 12796Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2965Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10312m 34% 9485Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1802m 6% 4846Mi 4% gke-xlou-cdm-ds-0f601751-j488 9003m 30% 10344Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16160m 54% 10929Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 10359m 34% 7139Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8160m 51% 8269Mi 14% 22:33:07 DEBUG --- stderr --- 22:33:07 DEBUG 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "1335.1333333333332" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 ERROR [http_cmd]: ERROR 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 ERROR --- status code --- 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG --- status code --- 22:34:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 ERROR --- http response --- 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG "resultType": "vector", 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG "result": [ 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG "result": [ 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG { 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG "metric": { 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG "metric": { 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG "instance": "10.142.0.107:9100" 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:34:05 DEBUG "instance": "10.142.0.107:9100" 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG }, 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG }, 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG "value": [ 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:34:05 DEBUG }, 22:34:05 DEBUG { 22:34:05 DEBUG "value": [ 22:34:05 DEBUG { 22:34:05 DEBUG 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG { 22:34:05 DEBUG }, 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG "value": [ 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "value": [ 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG 22:34:05 DEBUG "data": { 22:34:05 DEBUG 22:34:05 DEBUG "data": { 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [...] 22:34:05 DEBUG "data": { 22:34:05 DEBUG 22:34:05 DEBUG { 22:34:05 DEBUG 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:05 DEBUG 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG 22:34:05 DEBUG "result": [ 22:34:05 DEBUG [...] 22:34:05 DEBUG "result": [ 22:34:05 DEBUG 22:34:05 DEBUG "result": [ 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG 22:34:05 DEBUG "data": { 22:34:05 DEBUG [...] 22:34:05 DEBUG { 22:34:05 DEBUG 22:34:05 DEBUG { 22:34:05 DEBUG 22:34:05 DEBUG { 22:34:05 DEBUG { 22:34:05 DEBUG [...] 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG 22:34:05 DEBUG "metric": { 22:34:05 DEBUG 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "instance": "10.142.0.43:9100" 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG 22:34:05 DEBUG "result": [ 22:34:05 DEBUG 22:34:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:34:05 DEBUG 22:34:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:34:05 DEBUG }, 22:34:05 DEBUG "pod": "ds-idrepo-2" 22:34:05 DEBUG "data": { 22:34:05 DEBUG 22:34:05 DEBUG { 22:34:05 DEBUG 22:34:05 DEBUG }, 22:34:05 DEBUG "instance": "10.142.0.43:9100" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG }, 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:34:05 DEBUG "value": [ 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG "result": [ 22:34:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:34:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:34:05 DEBUG }, 22:34:05 DEBUG 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG "12629.099999999997" 22:34:05 DEBUG 22:34:05 DEBUG { 22:34:05 DEBUG }, 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG 22:34:05 DEBUG ] 22:34:05 DEBUG 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "value": [ 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG 22:34:05 DEBUG "0.000500000000000019" 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG "pod": "ds-idrepo-2" 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG 22:34:05 DEBUG "6216007680" 22:34:05 DEBUG [...] 22:34:05 DEBUG ] 22:34:05 DEBUG [...] 22:34:05 DEBUG ] 22:34:05 DEBUG [...] 22:34:05 DEBUG }, 22:34:05 DEBUG "0.08" 22:34:05 DEBUG 22:34:05 DEBUG ] 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG "value": [ 22:34:05 DEBUG ] 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG ] 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG [...] 22:34:05 DEBUG ] 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG ] 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:34:05 DEBUG } 22:34:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:34:05 DEBUG "pod": "ds-idrepo-0" 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG } 22:34:05 DEBUG }, 22:34:05 DEBUG }, 22:34:05 DEBUG }, 22:34:05 DEBUG [...] 22:34:05 DEBUG } 22:34:05 DEBUG 22:34:05 DEBUG "value": [ 22:34:05 DEBUG "value": [ 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG 22:34:05 DEBUG }, 22:34:05 DEBUG "0" 22:34:05 DEBUG "57.9" 22:34:05 DEBUG "0" 22:34:05 DEBUG 22:34:05 DEBUG "value": [ 22:34:05 DEBUG ] 22:34:05 DEBUG ] 22:34:05 DEBUG ] 22:34:05 DEBUG "pod": "ds-cts-2" 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG }, 22:34:05 DEBUG "8.023730806788102" 22:34:05 DEBUG ] 22:34:05 DEBUG ] 22:34:05 DEBUG ] 22:34:05 DEBUG "value": [ 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "1335.2666666666667" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "ds-idrepo-2" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "ds-idrepo-0" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "316455" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "1335.3" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "instance": "10.142.0.107:9100" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "instance": "10.142.0.43:9100" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "4896.366666666666" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "ds-idrepo-2" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "ds-idrepo-0" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "ds-idrepo-2" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "ds-idrepo-0" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "ds-idrepo-2" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "ds-idrepo-0" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "1941385.7182410324" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "instance": "10.142.0.107:9100" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "instance": "10.142.0.43:9100" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0.00033333333333333365" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216845" 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "ds-idrepo-2" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "ds-idrepo-0" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:05 INFO [http_cmd]: http status code OK 22:34:05 DEBUG --- status code --- 22:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:05 DEBUG --- http response --- 22:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:05 DEBUG ----- output ----- 22:34:05 DEBUG { 22:34:05 DEBUG "status": "success", 22:34:05 DEBUG "data": { 22:34:05 DEBUG "resultType": "vector", 22:34:05 DEBUG "result": [ 22:34:05 DEBUG { 22:34:05 DEBUG "metric": { 22:34:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG [...] 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG 22:34:05 DEBUG "pod": "lodemon-58b6f5b7dd-dq2sv" 22:34:05 DEBUG }, 22:34:05 DEBUG "value": [ 22:34:05 DEBUG 1713216845, 22:34:05 DEBUG "0.0009213295887739009" 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG ] 22:34:05 DEBUG } 22:34:05 DEBUG } 22:34:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216846" 22:34:06 INFO [http_cmd]: http status code OK 22:34:06 DEBUG --- status code --- 22:34:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:06 DEBUG --- http response --- 22:34:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:06 DEBUG ----- output ----- 22:34:06 DEBUG { 22:34:06 DEBUG "status": "success", 22:34:06 DEBUG "data": { 22:34:06 DEBUG "resultType": "vector", 22:34:06 DEBUG "result": [ 22:34:06 DEBUG { 22:34:06 DEBUG "metric": { 22:34:06 DEBUG "instance": "10.142.0.107:9100" 22:34:06 DEBUG }, 22:34:06 DEBUG "value": [ 22:34:06 DEBUG 22:34:06 DEBUG 22:34:06 DEBUG 22:34:06 DEBUG [...] 22:34:06 DEBUG 22:34:06 DEBUG 22:34:06 DEBUG 22:34:06 DEBUG "instance": "10.142.0.43:9100" 22:34:06 DEBUG }, 22:34:06 DEBUG "value": [ 22:34:06 DEBUG 1713216846, 22:34:06 DEBUG "13875596288" 22:34:06 DEBUG ] 22:34:06 DEBUG } 22:34:06 DEBUG ] 22:34:06 DEBUG } 22:34:06 DEBUG } 22:34:06 INFO 22:34:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:34:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:34:06 INFO [loop_until]: OK (rc = 0) 22:34:06 DEBUG --- stdout --- 22:34:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5647m 8666Mi am-7d597cf7d6-8vr8f 5440m 7994Mi am-7d597cf7d6-9wz9h 7307m 9233Mi ds-cts-0 3790m 8487Mi ds-cts-1 2694m 8579Mi ds-cts-2 3002m 8729Mi ds-idrepo-0 6882m 4956Mi ds-idrepo-1 1033m 2609Mi ds-idrepo-2 1036m 2599Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7896m 6092Mi 22:34:06 DEBUG --- stderr --- 22:34:06 DEBUG 22:34:07 INFO 22:34:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:34:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:34:07 INFO [loop_until]: OK (rc = 0) 22:34:07 DEBUG --- stdout --- 22:34:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5075Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3316m 11% 10850Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6072m 20% 10740Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11715m 39% 3653Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1105m 3% 5435Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4601m 15% 12822Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2971Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5988m 20% 9474Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1068m 3% 4850Mi 4% gke-xlou-cdm-ds-0f601751-j488 3171m 10% 10357Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2461Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 7605m 25% 10893Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7215m 24% 7137Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8212m 51% 9646Mi 16% 22:34:07 DEBUG --- stderr --- 22:34:07 DEBUG 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG { 22:35:05 DEBUG { 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "data": { 22:35:05 DEBUG "data": { 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG "result": [ 22:35:05 DEBUG "result": [ 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG { 22:35:05 DEBUG { 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "ds-idrepo-2" 22:35:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:35:05 DEBUG "pod": "ds-idrepo-2" 22:35:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:35:05 DEBUG }, 22:35:05 DEBUG }, 22:35:05 DEBUG }, 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG "value": [ 22:35:05 DEBUG "value": [ 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG [...] 22:35:05 DEBUG [...] 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG "pod": "ds-cts-2" 22:35:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:35:05 DEBUG "pod": "ds-idrepo-0" 22:35:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG }, 22:35:05 DEBUG }, 22:35:05 DEBUG }, 22:35:05 DEBUG }, 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG "value": [ 22:35:05 DEBUG "value": [ 22:35:05 DEBUG "value": [ 22:35:05 DEBUG "value": [ 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0" 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG "0" 22:35:05 DEBUG "0" 22:35:05 DEBUG "2395.5" 22:35:05 DEBUG ] 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG ] 22:35:05 DEBUG ] 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG { 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG ] 22:35:05 DEBUG ] 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG "data": { 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG "result": [ 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG { 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG "metric": { 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0.08" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ] 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG } 22:35:05 DEBUG { 22:35:05 DEBUG } 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "8.032453078916863" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216905" 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 ERROR [http_cmd]: ERROR 22:35:05 DEBUG ----- output ----- 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 ERROR --- status code --- 22:35:05 DEBUG { 22:35:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:35:05 DEBUG "status": "success", 22:35:05 ERROR --- http response --- 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "1316.8333333333333" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:35:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "instance": "10.142.0.107:9100" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "instance": "10.142.0.43:9100" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0.0002666666666669926" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "instance": "10.142.0.107:9100" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "instance": "10.142.0.43:9100" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "12713.766666666665" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "8250454016" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "ds-idrepo-2" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "ds-idrepo-0" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "1316.6333333333332" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "308767.5" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "1316.9333333333334" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "ds-cts-1" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "instance": "10.142.0.107:9100" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "instance": "10.142.0.43:9100" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "3066.8" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "ds-idrepo-2" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "ds-idrepo-0" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "ds-cts-1" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216905" 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "ds-idrepo-2" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "ds-idrepo-0" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:05 INFO [http_cmd]: http status code OK 22:35:05 DEBUG --- status code --- 22:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:05 DEBUG --- http response --- 22:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:05 DEBUG ----- output ----- 22:35:05 DEBUG { 22:35:05 DEBUG "status": "success", 22:35:05 DEBUG "data": { 22:35:05 DEBUG "resultType": "vector", 22:35:05 DEBUG "result": [ 22:35:05 DEBUG { 22:35:05 DEBUG "metric": { 22:35:05 DEBUG "pod": "ds-idrepo-2" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG [...] 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG 22:35:05 DEBUG "pod": "ds-idrepo-0" 22:35:05 DEBUG }, 22:35:05 DEBUG "value": [ 22:35:05 DEBUG 1713216905, 22:35:05 DEBUG "0" 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG ] 22:35:05 DEBUG } 22:35:05 DEBUG } 22:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216906" 22:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216906" 22:35:06 INFO [http_cmd]: http status code OK 22:35:06 DEBUG --- status code --- 22:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:06 DEBUG --- http response --- 22:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:35:06 DEBUG ----- output ----- 22:35:06 DEBUG { 22:35:06 DEBUG "status": "success", 22:35:06 DEBUG "data": { 22:35:06 DEBUG "resultType": "vector", 22:35:06 DEBUG "result": [ 22:35:06 DEBUG { 22:35:06 DEBUG "metric": { 22:35:06 DEBUG "pod": "ds-cts-1" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG [...] 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 1713216906, 22:35:06 DEBUG "0.000025012025012004837" 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG } 22:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216906" 22:35:06 INFO [http_cmd]: http status code OK 22:35:06 DEBUG --- status code --- 22:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:06 DEBUG --- http response --- 22:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:35:06 DEBUG ----- output ----- 22:35:06 DEBUG { 22:35:06 DEBUG "status": "success", 22:35:06 DEBUG "data": { 22:35:06 DEBUG "resultType": "vector", 22:35:06 DEBUG "result": [ 22:35:06 DEBUG { 22:35:06 DEBUG "metric": { 22:35:06 DEBUG "pod": "ds-cts-1" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG [...] 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 1713216906, 22:35:06 DEBUG "0" 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG } 22:35:06 INFO [http_cmd]: http status code OK 22:35:06 DEBUG --- status code --- 22:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:06 DEBUG --- http response --- 22:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:06 DEBUG ----- output ----- 22:35:06 DEBUG { 22:35:06 DEBUG "status": "success", 22:35:06 DEBUG "data": { 22:35:06 DEBUG "resultType": "vector", 22:35:06 DEBUG "result": [ 22:35:06 DEBUG { 22:35:06 DEBUG "metric": { 22:35:06 DEBUG "instance": "10.142.0.107:9100" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG [...] 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG "instance": "10.142.0.43:9100" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 1713216906, 22:35:06 DEBUG "0.0003333333333333262" 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG } 22:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216906" 22:35:06 INFO [http_cmd]: http status code OK 22:35:06 DEBUG --- status code --- 22:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:06 DEBUG --- http response --- 22:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:06 DEBUG ----- output ----- 22:35:06 DEBUG { 22:35:06 DEBUG "status": "success", 22:35:06 DEBUG "data": { 22:35:06 DEBUG "resultType": "vector", 22:35:06 DEBUG "result": [ 22:35:06 DEBUG { 22:35:06 DEBUG "metric": { 22:35:06 DEBUG "pod": "ds-idrepo-2" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG [...] 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG "pod": "ds-idrepo-0" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 1713216906, 22:35:06 DEBUG "0" 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG } 22:35:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216906" 22:35:06 INFO [http_cmd]: http status code OK 22:35:06 DEBUG --- status code --- 22:35:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:06 DEBUG --- http response --- 22:35:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:06 DEBUG ----- output ----- 22:35:06 DEBUG { 22:35:06 DEBUG "status": "success", 22:35:06 DEBUG "data": { 22:35:06 DEBUG "resultType": "vector", 22:35:06 DEBUG "result": [ 22:35:06 DEBUG { 22:35:06 DEBUG "metric": { 22:35:06 DEBUG "instance": "10.142.0.107:9100" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG [...] 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG 22:35:06 DEBUG "instance": "10.142.0.43:9100" 22:35:06 DEBUG }, 22:35:06 DEBUG "value": [ 22:35:06 DEBUG 1713216906, 22:35:06 DEBUG "13876610048" 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG ] 22:35:06 DEBUG } 22:35:06 DEBUG } 22:35:06 INFO 22:35:06 INFO [loop_until]: kubectl --namespace=xlou top pods 22:35:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:35:07 INFO [loop_until]: OK (rc = 0) 22:35:07 DEBUG --- stdout --- 22:35:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 8086m 8668Mi am-7d597cf7d6-8vr8f 7841m 8001Mi am-7d597cf7d6-9wz9h 10384m 9058Mi ds-cts-0 10954m 8502Mi ds-cts-1 8853m 8558Mi ds-cts-2 8540m 8722Mi ds-idrepo-0 8281m 4956Mi ds-idrepo-1 1178m 2609Mi ds-idrepo-2 1196m 2601Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8005m 7851Mi 22:35:07 DEBUG --- stderr --- 22:35:07 DEBUG 22:35:07 INFO 22:35:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:35:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:35:07 INFO [loop_until]: OK (rc = 0) 22:35:07 DEBUG --- stdout --- 22:35:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5063Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8691m 29% 10825Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8617m 28% 10755Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11247m 37% 3655Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1241m 4% 5447Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 12911m 43% 12815Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2962Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8556m 28% 9500Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1332m 4% 4834Mi 4% gke-xlou-cdm-ds-0f601751-j488 8433m 28% 10346Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2457Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 11291m 37% 10703Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 8637m 28% 7132Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8238m 51% 11118Mi 18% 22:35:07 DEBUG --- stderr --- 22:35:07 DEBUG 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "0.08" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "ds-idrepo-2" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "ds-cts-2" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "0" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "instance": "10.142.0.107:9100" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "instance": "10.142.0.43:9100" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "0.00040000000000001514" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 ERROR [http_cmd]: ERROR 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:05 ERROR --- status code --- 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG ----- output ----- 22:36:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG { 22:36:05 ERROR --- http response --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:36:05 DEBUG --- http response --- 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216965" 22:36:05 DEBUG "data": { 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "data": { 22:36:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG "result": [ 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG "result": [ 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG { 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG { 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG "metric": { 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "metric": { 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "data": { 22:36:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG }, 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG "pod": "ds-idrepo-2" 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG }, 22:36:05 DEBUG { 22:36:05 DEBUG "value": [ 22:36:05 DEBUG { 22:36:05 DEBUG }, 22:36:05 DEBUG "result": [ 22:36:05 DEBUG "value": [ 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "value": [ 22:36:05 DEBUG { 22:36:05 DEBUG 22:36:05 DEBUG "data": { 22:36:05 DEBUG 22:36:05 DEBUG "data": { 22:36:05 DEBUG 22:36:05 DEBUG "metric": { 22:36:05 DEBUG 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG 22:36:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:36:05 DEBUG 22:36:05 DEBUG "result": [ 22:36:05 DEBUG [...] 22:36:05 DEBUG "result": [ 22:36:05 DEBUG 22:36:05 DEBUG }, 22:36:05 DEBUG [...] 22:36:05 DEBUG { 22:36:05 DEBUG 22:36:05 DEBUG { 22:36:05 DEBUG [...] 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG "metric": { 22:36:05 DEBUG 22:36:05 DEBUG "metric": { 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:36:05 DEBUG 22:36:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG }, 22:36:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:36:05 DEBUG }, 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:36:05 DEBUG "value": [ 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG "pod": "ds-idrepo-0" 22:36:05 DEBUG [...] 22:36:05 DEBUG }, 22:36:05 DEBUG 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG }, 22:36:05 DEBUG 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG 22:36:05 DEBUG "2027.8666666666666" 22:36:05 DEBUG 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG 22:36:05 DEBUG "0" 22:36:05 DEBUG [...] 22:36:05 DEBUG ] 22:36:05 DEBUG [...] 22:36:05 DEBUG "0" 22:36:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:36:05 DEBUG ] 22:36:05 DEBUG 22:36:05 DEBUG } 22:36:05 DEBUG 22:36:05 DEBUG ] 22:36:05 DEBUG }, 22:36:05 DEBUG } 22:36:05 DEBUG 22:36:05 DEBUG ] 22:36:05 DEBUG 22:36:05 DEBUG } 22:36:05 DEBUG "value": [ 22:36:05 DEBUG ] 22:36:05 DEBUG 22:36:05 DEBUG } 22:36:05 DEBUG 22:36:05 DEBUG ] 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG } 22:36:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:36:05 DEBUG } 22:36:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:36:05 DEBUG } 22:36:05 DEBUG "1269.2" 22:36:05 DEBUG } 22:36:05 DEBUG }, 22:36:05 DEBUG }, 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 DEBUG "value": [ 22:36:05 DEBUG "value": [ 22:36:05 DEBUG } 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG ] 22:36:05 DEBUG "8.030083218996225" 22:36:05 DEBUG "8416047104" 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "instance": "10.142.0.107:9100" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "instance": "10.142.0.43:9100" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "12523.899999999998" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG }, 22:36:05 DEBUG { 22:36:05 DEBUG "value": [ 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG 22:36:05 DEBUG "data": { 22:36:05 DEBUG 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG 22:36:05 DEBUG "result": [ 22:36:05 DEBUG [...] 22:36:05 DEBUG { 22:36:05 DEBUG 22:36:05 DEBUG "metric": { 22:36:05 DEBUG 22:36:05 DEBUG "pod": "ds-idrepo-2" 22:36:05 DEBUG 22:36:05 DEBUG }, 22:36:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:36:05 DEBUG "value": [ 22:36:05 DEBUG }, 22:36:05 DEBUG 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG 22:36:05 DEBUG "1268.4666666666667" 22:36:05 DEBUG [...] 22:36:05 DEBUG ] 22:36:05 DEBUG 22:36:05 DEBUG } 22:36:05 DEBUG 22:36:05 DEBUG ] 22:36:05 DEBUG 22:36:05 DEBUG } 22:36:05 DEBUG "pod": "ds-idrepo-0" 22:36:05 DEBUG } 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "0" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "248387.5" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "1268.3" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "0" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216965" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "instance": "10.142.0.107:9100" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "instance": "10.142.0.43:9100" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "4684.466666666666" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216965" 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "ds-idrepo-2" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "ds-idrepo-0" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "0" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:05 INFO [http_cmd]: http status code OK 22:36:05 DEBUG --- status code --- 22:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:05 DEBUG --- http response --- 22:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:36:05 DEBUG ----- output ----- 22:36:05 DEBUG { 22:36:05 DEBUG "status": "success", 22:36:05 DEBUG "data": { 22:36:05 DEBUG "resultType": "vector", 22:36:05 DEBUG "result": [ 22:36:05 DEBUG { 22:36:05 DEBUG "metric": { 22:36:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG [...] 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG 22:36:05 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 22:36:05 DEBUG }, 22:36:05 DEBUG "value": [ 22:36:05 DEBUG 1713216965, 22:36:05 DEBUG "0" 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG ] 22:36:05 DEBUG } 22:36:05 DEBUG } 22:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216966" 22:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216966" 22:36:06 INFO [http_cmd]: http status code OK 22:36:06 DEBUG --- status code --- 22:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:06 DEBUG --- http response --- 22:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:06 DEBUG ----- output ----- 22:36:06 DEBUG { 22:36:06 DEBUG "status": "success", 22:36:06 DEBUG "data": { 22:36:06 DEBUG "resultType": "vector", 22:36:06 DEBUG "result": [ 22:36:06 DEBUG { 22:36:06 DEBUG "metric": { 22:36:06 DEBUG "pod": "ds-idrepo-2" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG [...] 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG "pod": "ds-idrepo-0" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 1713216966, 22:36:06 DEBUG "0" 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG } 22:36:06 INFO [http_cmd]: http status code OK 22:36:06 DEBUG --- status code --- 22:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:06 DEBUG --- http response --- 22:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:06 DEBUG ----- output ----- 22:36:06 DEBUG { 22:36:06 DEBUG "status": "success", 22:36:06 DEBUG "data": { 22:36:06 DEBUG "resultType": "vector", 22:36:06 DEBUG "result": [ 22:36:06 DEBUG { 22:36:06 DEBUG "metric": { 22:36:06 DEBUG "pod": "ds-idrepo-2" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG [...] 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG "pod": "ds-idrepo-0" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 1713216966, 22:36:06 DEBUG "0" 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG } 22:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216966" 22:36:06 INFO [http_cmd]: http status code OK 22:36:06 DEBUG --- status code --- 22:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:06 DEBUG --- http response --- 22:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:06 DEBUG ----- output ----- 22:36:06 DEBUG { 22:36:06 DEBUG "status": "success", 22:36:06 DEBUG "data": { 22:36:06 DEBUG "resultType": "vector", 22:36:06 DEBUG "result": [ 22:36:06 DEBUG { 22:36:06 DEBUG "metric": { 22:36:06 DEBUG "pod": "ds-idrepo-2" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG [...] 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG "pod": "ds-idrepo-0" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 1713216966, 22:36:06 DEBUG "0" 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG } 22:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216966" 22:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713216966" 22:36:06 INFO [http_cmd]: http status code OK 22:36:06 INFO [http_cmd]: http status code OK 22:36:06 DEBUG --- status code --- 22:36:06 DEBUG --- status code --- 22:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:06 DEBUG --- http response --- 22:36:06 DEBUG --- http response --- 22:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:36:06 DEBUG ----- output ----- 22:36:06 DEBUG { 22:36:06 DEBUG "status": "success", 22:36:06 DEBUG "data": { 22:36:06 DEBUG "resultType": "vector", 22:36:06 DEBUG "result": [ 22:36:06 DEBUG { 22:36:06 DEBUG "metric": { 22:36:06 DEBUG "pod": "ds-cts-1" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG [...] 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 1713216966, 22:36:06 DEBUG "0" 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG } 22:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:36:06 DEBUG ----- output ----- 22:36:06 DEBUG { 22:36:06 DEBUG "status": "success", 22:36:06 DEBUG "data": { 22:36:06 DEBUG "resultType": "vector", 22:36:06 DEBUG "result": [ 22:36:06 DEBUG { 22:36:06 DEBUG "metric": { 22:36:06 DEBUG "pod": "ds-cts-1" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG [...] 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 1713216966, 22:36:06 DEBUG "0.000022947442953242112" 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG } 22:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216966" 22:36:06 INFO [http_cmd]: http status code OK 22:36:06 DEBUG --- status code --- 22:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:06 DEBUG --- http response --- 22:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:06 DEBUG ----- output ----- 22:36:06 DEBUG { 22:36:06 DEBUG "status": "success", 22:36:06 DEBUG "data": { 22:36:06 DEBUG "resultType": "vector", 22:36:06 DEBUG "result": [ 22:36:06 DEBUG { 22:36:06 DEBUG "metric": { 22:36:06 DEBUG "instance": "10.142.0.107:9100" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG [...] 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG "instance": "10.142.0.43:9100" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 1713216966, 22:36:06 DEBUG "0.0006666666666666598" 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG } 22:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713216966" 22:36:06 INFO [http_cmd]: http status code OK 22:36:06 DEBUG --- status code --- 22:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:06 DEBUG --- http response --- 22:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:06 DEBUG ----- output ----- 22:36:06 DEBUG { 22:36:06 DEBUG "status": "success", 22:36:06 DEBUG "data": { 22:36:06 DEBUG "resultType": "vector", 22:36:06 DEBUG "result": [ 22:36:06 DEBUG { 22:36:06 DEBUG "metric": { 22:36:06 DEBUG "instance": "10.142.0.107:9100" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG [...] 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG 22:36:06 DEBUG "instance": "10.142.0.43:9100" 22:36:06 DEBUG }, 22:36:06 DEBUG "value": [ 22:36:06 DEBUG 1713216966, 22:36:06 DEBUG "13877079040" 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG ] 22:36:06 DEBUG } 22:36:06 DEBUG } 22:36:07 INFO 22:36:07 INFO [loop_until]: kubectl --namespace=xlou top pods 22:36:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:36:07 INFO [loop_until]: OK (rc = 0) 22:36:07 DEBUG --- stdout --- 22:36:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7438m 8670Mi am-7d597cf7d6-8vr8f 6991m 9056Mi am-7d597cf7d6-9wz9h 9466m 9008Mi ds-cts-0 9746m 8498Mi ds-cts-1 6602m 8569Mi ds-cts-2 7017m 8733Mi ds-idrepo-0 7637m 4956Mi ds-idrepo-1 1098m 2609Mi ds-idrepo-2 1108m 2601Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7818m 7839Mi 22:36:07 DEBUG --- stderr --- 22:36:07 DEBUG 22:36:07 INFO 22:36:07 INFO [loop_until]: kubectl --namespace=xlou top node 22:36:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:36:07 INFO [loop_until]: OK (rc = 0) 22:36:07 DEBUG --- stdout --- 22:36:07 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5063Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6500m 21% 10820Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7401m 24% 10751Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11110m 37% 3657Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1152m 3% 5457Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9957m 33% 12804Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2963Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7412m 24% 10562Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1225m 4% 4838Mi 4% gke-xlou-cdm-ds-0f601751-j488 6378m 21% 10348Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2462Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 10455m 35% 10675Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7922m 26% 7143Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8198m 51% 11107Mi 18% 22:36:07 DEBUG --- stderr --- 22:36:07 DEBUG 22:36:54 WARNING File /tmp/lodemon_data-20240415_215803/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 22:36:54] "GET /monitoring/average?start_time=20240415_213222&stop_time=20240415_213453 HTTP/1.1" 200 - 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217025" 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG --- http response --- 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713217025" 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG ----- output ----- 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG { 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:05 DEBUG "data": { 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG { 22:37:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:37:05 DEBUG "result": [ 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG { 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG { 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG { 22:37:05 DEBUG "data": { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "data": { 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG "data": { 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG "result": [ 22:37:05 DEBUG }, 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG { 22:37:05 DEBUG { 22:37:05 DEBUG "value": [ 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "metric": { 22:37:05 DEBUG 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "data": { 22:37:05 DEBUG { 22:37:05 DEBUG "data": { 22:37:05 DEBUG "pod": "ds-idrepo-2" 22:37:05 DEBUG 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG "instance": "10.142.0.107:9100" 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG }, 22:37:05 DEBUG 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG }, 22:37:05 DEBUG "result": [ 22:37:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:37:05 DEBUG "result": [ 22:37:05 DEBUG "value": [ 22:37:05 DEBUG [...] 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG "value": [ 22:37:05 DEBUG { 22:37:05 DEBUG }, 22:37:05 DEBUG { 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "value": [ 22:37:05 DEBUG "metric": { 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:05 DEBUG "pod": "ds-idrepo-2" 22:37:05 DEBUG 22:37:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG }, 22:37:05 DEBUG 22:37:05 DEBUG }, 22:37:05 DEBUG [...] 22:37:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:05 DEBUG [...] 22:37:05 DEBUG { 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG }, 22:37:05 DEBUG 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG "data": { 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "ds-cts-2" 22:37:05 DEBUG "0.08" 22:37:05 DEBUG "instance": "10.142.0.43:9100" 22:37:05 DEBUG "result": [ 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG }, 22:37:05 DEBUG ] 22:37:05 DEBUG }, 22:37:05 DEBUG { 22:37:05 DEBUG 22:37:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:05 DEBUG 22:37:05 DEBUG "value": [ 22:37:05 DEBUG } 22:37:05 DEBUG "value": [ 22:37:05 DEBUG "metric": { 22:37:05 DEBUG 22:37:05 DEBUG }, 22:37:05 DEBUG 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG ] 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "instance": "10.142.0.107:9100" 22:37:05 DEBUG 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG "0" 22:37:05 DEBUG } 22:37:05 DEBUG "0.00030000000000001136" 22:37:05 DEBUG }, 22:37:05 DEBUG "pod": "ds-idrepo-0" 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG "value": [ 22:37:05 DEBUG }, 22:37:05 DEBUG "1337.2666666666667" 22:37:05 DEBUG }, 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG "value": [ 22:37:05 DEBUG ] 22:37:05 DEBUG "value": [ 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG ] 22:37:05 DEBUG ] 22:37:05 DEBUG 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG } 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG "0" 22:37:05 DEBUG ] 22:37:05 DEBUG "341.76666666666665" 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG [...] 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG ] 22:37:05 DEBUG ] 22:37:05 DEBUG { 22:37:05 DEBUG 22:37:05 DEBUG { 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "instance": "10.142.0.43:9100" 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG "data": { 22:37:05 DEBUG }, 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "value": [ 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "12290.33333333333" 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG ] 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:37:05 DEBUG } 22:37:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:37:05 DEBUG }, 22:37:05 DEBUG ] 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG } 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG } 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:37:05 DEBUG }, 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "0" 22:37:05 DEBUG "8.04992924702583" 22:37:05 DEBUG ] 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 ERROR [http_cmd]: ERROR 22:37:05 ERROR --- status code --- 22:37:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:37:05 ERROR --- http response --- 22:37:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:37:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "pod": "ds-idrepo-2" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "ds-idrepo-0" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "0" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713217025" 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "8636702720" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "340.3666666666667" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "220027.5" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "336.73333333333335" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217025" 22:37:05 INFO [http_cmd]: http status code OK 22:37:05 DEBUG --- status code --- 22:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:05 DEBUG --- http response --- 22:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:37:05 DEBUG ----- output ----- 22:37:05 DEBUG { 22:37:05 DEBUG "status": "success", 22:37:05 DEBUG "data": { 22:37:05 DEBUG "resultType": "vector", 22:37:05 DEBUG "result": [ 22:37:05 DEBUG { 22:37:05 DEBUG "metric": { 22:37:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG [...] 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG 22:37:05 DEBUG "pod": "ds-cts-2" 22:37:05 DEBUG }, 22:37:05 DEBUG "value": [ 22:37:05 DEBUG 1713217025, 22:37:05 DEBUG "17377757.07154742" 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG ] 22:37:05 DEBUG } 22:37:05 DEBUG } 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217026" 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "instance": "10.142.0.107:9100" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "instance": "10.142.0.43:9100" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "2960.8333333333335" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217026" 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217026" 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "pod": "ds-idrepo-2" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "pod": "ds-idrepo-0" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "pod": "ds-cts-2" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217026" 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217026" 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "pod": "ds-idrepo-2" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "pod": "ds-idrepo-0" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "pod": "ds-idrepo-2" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "pod": "ds-idrepo-0" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217026" 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "pod": "ds-idrepo-2" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "pod": "ds-idrepo-0" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217026" 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217026" 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217026" 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "instance": "10.142.0.107:9100" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "instance": "10.142.0.43:9100" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "0" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217026" 22:37:06 INFO [http_cmd]: http status code OK 22:37:06 DEBUG --- status code --- 22:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:06 DEBUG --- http response --- 22:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:06 DEBUG ----- output ----- 22:37:06 DEBUG { 22:37:06 DEBUG "status": "success", 22:37:06 DEBUG "data": { 22:37:06 DEBUG "resultType": "vector", 22:37:06 DEBUG "result": [ 22:37:06 DEBUG { 22:37:06 DEBUG "metric": { 22:37:06 DEBUG "instance": "10.142.0.107:9100" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG [...] 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG 22:37:06 DEBUG "instance": "10.142.0.43:9100" 22:37:06 DEBUG }, 22:37:06 DEBUG "value": [ 22:37:06 DEBUG 1713217026, 22:37:06 DEBUG "13877728256" 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG ] 22:37:06 DEBUG } 22:37:06 DEBUG } 22:37:07 INFO 22:37:07 INFO [loop_until]: kubectl --namespace=xlou top pods 22:37:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:37:07 INFO [loop_until]: OK (rc = 0) 22:37:07 DEBUG --- stdout --- 22:37:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2932m 8668Mi am-7d597cf7d6-8vr8f 2930m 9053Mi am-7d597cf7d6-9wz9h 4350m 9018Mi ds-cts-0 4544m 8469Mi ds-cts-1 3960m 8544Mi ds-cts-2 3374m 8746Mi ds-idrepo-0 2503m 4956Mi ds-idrepo-1 292m 2617Mi ds-idrepo-2 433m 2600Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 2235m 16Mi 22:37:07 DEBUG --- stderr --- 22:37:07 DEBUG 22:37:08 INFO 22:37:08 INFO [loop_until]: kubectl --namespace=xlou top node 22:37:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:37:08 INFO [loop_until]: OK (rc = 0) 22:37:08 DEBUG --- stdout --- 22:37:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5076Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2985m 9% 10813Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1829m 6% 10773Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 1318m 4% 3116Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 264m 0% 5466Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4568m 15% 12812Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2959Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2459m 8% 10566Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 479m 1% 4862Mi 4% gke-xlou-cdm-ds-0f601751-j488 2963m 9% 10374Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2460Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 4107m 13% 10689Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 945m 3% 7142Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1971m 12% 3921Mi 6% 22:37:08 DEBUG --- stderr --- 22:37:08 DEBUG 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713217085" 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "pod": "ds-idrepo-2" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "pod": "ds-cts-2" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG "0" 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG --- status code --- 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 ERROR [http_cmd]: ERROR 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 ERROR --- status code --- 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 ERROR --- http response --- 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:38:05 DEBUG { 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:05 DEBUG { 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:05 DEBUG "status": "success", 22:38:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG "data": { 22:38:05 DEBUG "data": { 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG { 22:38:05 DEBUG "data": { 22:38:05 DEBUG { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG "result": [ 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG "data": { 22:38:05 DEBUG "result": [ 22:38:05 DEBUG "data": { 22:38:05 DEBUG { 22:38:05 DEBUG { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "pod": "ds-idrepo-2" 22:38:05 DEBUG "result": [ 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "result": [ 22:38:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:38:05 DEBUG "pod": "ds-idrepo-2" 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:38:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:38:05 DEBUG }, 22:38:05 DEBUG { 22:38:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:38:05 DEBUG { 22:38:05 DEBUG }, 22:38:05 DEBUG }, 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:38:05 DEBUG }, 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "metric": { 22:38:05 DEBUG }, 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "instance": "10.142.0.107:9100" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG "instance": "10.142.0.107:9100" 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG }, 22:38:05 DEBUG 22:38:05 DEBUG }, 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "pod": "ds-idrepo-0" 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:05 DEBUG "pod": "ds-idrepo-0" 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:05 DEBUG }, 22:38:05 DEBUG 22:38:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:05 DEBUG 22:38:05 DEBUG }, 22:38:05 DEBUG }, 22:38:05 DEBUG 22:38:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:05 DEBUG }, 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG }, 22:38:05 DEBUG 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "instance": "10.142.0.43:9100" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG "instance": "10.142.0.43:9100" 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG "0" 22:38:05 DEBUG }, 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG }, 22:38:05 DEBUG "0.06666666666666667" 22:38:05 DEBUG "0" 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG "0" 22:38:05 DEBUG "0.08" 22:38:05 DEBUG ] 22:38:05 DEBUG "value": [ 22:38:05 DEBUG "1314.4" 22:38:05 DEBUG "value": [ 22:38:05 DEBUG ] 22:38:05 DEBUG ] 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG "0.16666666666666666" 22:38:05 DEBUG ] 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG ] 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG "660895.5333333332" 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG "0.0005333333333335114" 22:38:05 DEBUG } 22:38:05 DEBUG "0.028043977431449343" 22:38:05 DEBUG ] 22:38:05 DEBUG ] 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG ] 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217085" 22:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713217085" 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG "145572" 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:05 INFO [http_cmd]: http status code OK 22:38:05 DEBUG --- status code --- 22:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:05 DEBUG --- http response --- 22:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:05 DEBUG ----- output ----- 22:38:05 DEBUG { 22:38:05 DEBUG "status": "success", 22:38:05 DEBUG "data": { 22:38:05 DEBUG "resultType": "vector", 22:38:05 DEBUG "result": [ 22:38:05 DEBUG { 22:38:05 DEBUG "metric": { 22:38:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG [...] 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG 22:38:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:38:05 DEBUG }, 22:38:05 DEBUG "value": [ 22:38:05 DEBUG 1713217085, 22:38:05 DEBUG "114032640" 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG ] 22:38:05 DEBUG } 22:38:05 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0.06666666666666667" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "8300.559891790583" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "ds-idrepo-2" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "ds-idrepo-0" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "instance": "10.142.0.107:9100" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "instance": "10.142.0.43:9100" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "28020.26666666667" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "80744660.38918228" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "ds-idrepo-2" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "ds-idrepo-0" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "ds-idrepo-2" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "ds-idrepo-0" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG { 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG "data": { 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "ds-idrepo-2" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "ds-idrepo-0" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "instance": "10.142.0.107:9100" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "instance": "10.142.0.43:9100" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "0" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217086" 22:38:06 INFO [http_cmd]: http status code OK 22:38:06 DEBUG --- status code --- 22:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:06 DEBUG --- http response --- 22:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:06 DEBUG ----- output ----- 22:38:06 DEBUG { 22:38:06 DEBUG "status": "success", 22:38:06 DEBUG "data": { 22:38:06 DEBUG "resultType": "vector", 22:38:06 DEBUG "result": [ 22:38:06 DEBUG { 22:38:06 DEBUG "metric": { 22:38:06 DEBUG "instance": "10.142.0.107:9100" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG [...] 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG 22:38:06 DEBUG "instance": "10.142.0.43:9100" 22:38:06 DEBUG }, 22:38:06 DEBUG "value": [ 22:38:06 DEBUG 1713217086, 22:38:06 DEBUG "13878406144" 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG ] 22:38:06 DEBUG } 22:38:06 DEBUG } 22:38:07 INFO 22:38:07 INFO [loop_until]: kubectl --namespace=xlou top pods 22:38:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:38:07 INFO [loop_until]: OK (rc = 0) 22:38:07 DEBUG --- stdout --- 22:38:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1183m 8605Mi am-7d597cf7d6-8vr8f 1196m 9021Mi am-7d597cf7d6-9wz9h 1568m 8922Mi ds-cts-0 4003m 8476Mi ds-cts-1 2915m 8565Mi ds-cts-2 2785m 8758Mi ds-idrepo-0 1101m 4955Mi ds-idrepo-1 343m 2654Mi ds-idrepo-2 381m 2653Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 5m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 954m 335Mi 22:38:07 DEBUG --- stderr --- 22:38:07 DEBUG 22:38:08 INFO 22:38:08 INFO [loop_until]: kubectl --namespace=xlou top node 22:38:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:38:08 INFO [loop_until]: OK (rc = 0) 22:38:08 DEBUG --- stdout --- 22:38:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5086Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2862m 9% 10825Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1261m 4% 10722Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 92m 0% 3136Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 481m 1% 5471Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4299m 14% 12792Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 2982Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1321m 4% 10549Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 415m 1% 4834Mi 4% gke-xlou-cdm-ds-0f601751-j488 2957m 9% 10345Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 103m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1776m 5% 10616Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1079m 3% 7126Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 925m 5% 4168Mi 7% 22:38:08 DEBUG --- stderr --- 22:38:08 DEBUG 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG --- status code --- 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713217145" 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:05 DEBUG { 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 DEBUG ----- output ----- 22:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217145" 22:39:05 ERROR [http_cmd]: ERROR 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG { 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG { 22:39:05 ERROR --- status code --- 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG { 22:39:05 DEBUG "data": { 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:05 DEBUG { 22:39:05 DEBUG "status": "success", 22:39:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:39:05 DEBUG "data": { 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG --- status code --- 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG "data": { 22:39:05 ERROR --- http response --- 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG ----- output ----- 22:39:05 DEBUG "data": { 22:39:05 DEBUG "result": [ 22:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:05 DEBUG { 22:39:05 DEBUG { 22:39:05 DEBUG "data": { 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG "result": [ 22:39:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:39:05 DEBUG { 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG { 22:39:05 DEBUG --- http response --- 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG "result": [ 22:39:05 DEBUG { 22:39:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:39:05 DEBUG "status": "success", 22:39:05 DEBUG "result": [ 22:39:05 DEBUG "metric": { 22:39:05 DEBUG "data": { 22:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:05 DEBUG "data": { 22:39:05 DEBUG "result": [ 22:39:05 DEBUG { 22:39:05 DEBUG "metric": { 22:39:05 DEBUG "data": { 22:39:05 DEBUG { 22:39:06 DEBUG "metric": { 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG ----- output ----- 22:39:05 INFO [http_cmd]: http status code OK 22:39:05 DEBUG "resultType": "vector", 22:39:05 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG "pod": "ds-idrepo-2" 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG "result": [ 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG }, 22:39:05 DEBUG "instance": "10.142.0.107:9100" 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG { 22:39:06 DEBUG "instance": "10.142.0.107:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG "value": [ 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG }, 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:39:06 DEBUG }, 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "data": { 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG "metric": { 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG "value": [ 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "pod": "ds-idrepo-2" 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "result": [ 22:39:06 DEBUG }, 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG }, 22:39:06 DEBUG { 22:39:06 DEBUG "value": [ 22:39:06 DEBUG { 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG { 22:39:06 DEBUG 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "metric": { 22:39:06 DEBUG 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG [...] 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG [...] 22:39:06 DEBUG [...] 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG 22:39:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:39:06 DEBUG 22:39:06 DEBUG "data": { 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "data": { 22:39:06 DEBUG 22:39:06 DEBUG { 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG { 22:39:06 DEBUG 22:39:06 DEBUG }, 22:39:06 DEBUG 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG 22:39:06 DEBUG "value": [ 22:39:06 DEBUG [...] 22:39:06 DEBUG "result": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:39:06 DEBUG "result": [ 22:39:06 DEBUG 22:39:06 DEBUG "data": { 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "data": { 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG { 22:39:06 DEBUG 22:39:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:39:06 DEBUG }, 22:39:06 DEBUG { 22:39:06 DEBUG "instance": "10.142.0.43:9100" 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:39:06 DEBUG "pod": "ds-cts-2" 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "instance": "10.142.0.43:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "metric": { 22:39:06 DEBUG }, 22:39:06 DEBUG "result": [ 22:39:06 DEBUG }, 22:39:06 DEBUG }, 22:39:06 DEBUG "result": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:39:06 DEBUG "value": [ 22:39:06 DEBUG { 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "value": [ 22:39:06 DEBUG { 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "0.08" 22:39:06 DEBUG }, 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "metric": { 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:39:06 DEBUG 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "12279.833333333334" 22:39:06 DEBUG "pod": "ds-idrepo-2" 22:39:06 DEBUG "1270.2333333333333" 22:39:06 DEBUG "0" 22:39:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:39:06 DEBUG }, 22:39:06 DEBUG 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG "0.00026666666666651885" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG ] 22:39:06 DEBUG }, 22:39:06 DEBUG ] 22:39:06 DEBUG ] 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG "value": [ 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:39:06 DEBUG "0" 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG ] 22:39:06 DEBUG 22:39:06 DEBUG ] 22:39:06 DEBUG ] 22:39:06 DEBUG 22:39:06 DEBUG "841986048" 22:39:06 DEBUG }, 22:39:06 DEBUG ] 22:39:06 DEBUG [...] 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG [...] 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG ] 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "1.0324138169729846" 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG [...] 22:39:06 DEBUG ] 22:39:06 DEBUG ] 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG 22:39:06 DEBUG ] 22:39:06 DEBUG }, 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:39:06 DEBUG } 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG }, 22:39:06 DEBUG } 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "0" 22:39:06 DEBUG }, 22:39:06 DEBUG }, 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG ] 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "value": [ 22:39:06 DEBUG "68158.5" 22:39:06 DEBUG } 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG 1713217145, 22:39:06 DEBUG ] 22:39:06 DEBUG ] 22:39:06 DEBUG "0" 22:39:06 DEBUG "0" 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG ] 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG [...] 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG 22:39:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "instance": "10.142.0.107:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "instance": "10.142.0.43:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "4803" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-cts-1" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "10779.297573020565" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-idrepo-2" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-cts-1" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-idrepo-2" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-idrepo-2" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-idrepo-2" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-cts-1" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "1.1728472335039055" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "pod": "ds-cts-1" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "pod": "ds-idrepo-0" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "169603331.10237497" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "instance": "10.142.0.107:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "instance": "10.142.0.43:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "0" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217146" 22:39:06 INFO [http_cmd]: http status code OK 22:39:06 DEBUG --- status code --- 22:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:06 DEBUG --- http response --- 22:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:06 DEBUG ----- output ----- 22:39:06 DEBUG { 22:39:06 DEBUG "status": "success", 22:39:06 DEBUG "data": { 22:39:06 DEBUG "resultType": "vector", 22:39:06 DEBUG "result": [ 22:39:06 DEBUG { 22:39:06 DEBUG "metric": { 22:39:06 DEBUG "instance": "10.142.0.107:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG [...] 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG 22:39:06 DEBUG "instance": "10.142.0.43:9100" 22:39:06 DEBUG }, 22:39:06 DEBUG "value": [ 22:39:06 DEBUG 1713217146, 22:39:06 DEBUG "13879206912" 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG ] 22:39:06 DEBUG } 22:39:06 DEBUG } 22:39:07 INFO 22:39:07 INFO [loop_until]: kubectl --namespace=xlou top pods 22:39:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:39:07 INFO [loop_until]: OK (rc = 0) 22:39:07 DEBUG --- stdout --- 22:39:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1236m 8599Mi am-7d597cf7d6-8vr8f 1138m 9017Mi am-7d597cf7d6-9wz9h 1513m 8923Mi ds-cts-0 3681m 8480Mi ds-cts-1 2683m 8541Mi ds-cts-2 2586m 8719Mi ds-idrepo-0 1027m 4975Mi ds-idrepo-1 307m 2683Mi ds-idrepo-2 209m 2697Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1002m 1102Mi 22:39:07 DEBUG --- stderr --- 22:39:07 DEBUG 22:39:08 INFO 22:39:08 INFO [loop_until]: kubectl --namespace=xlou top node 22:39:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:39:08 INFO [loop_until]: OK (rc = 0) 22:39:08 DEBUG --- stdout --- 22:39:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5066Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2729m 9% 10816Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1222m 4% 10716Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3100Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 245m 0% 5527Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4101m 13% 12799Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2982Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1277m 4% 10520Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 277m 0% 4886Mi 4% gke-xlou-cdm-ds-0f601751-j488 2677m 8% 10358Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2463Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1705m 5% 10602Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1103m 3% 7150Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1083m 6% 4946Mi 8% 22:39:08 DEBUG --- stderr --- 22:39:08 DEBUG 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 ERROR [http_cmd]: ERROR 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 ERROR --- status code --- 22:40:06 DEBUG --- status code --- 22:40:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 ERROR --- http response --- 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:40:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:40:06 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG { 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "data": { 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG "result": [ 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG { 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "instance": "10.142.0.107:9100" 22:40:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:40:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:40:06 DEBUG }, 22:40:06 DEBUG }, 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG [...] 22:40:06 DEBUG [...] 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG "instance": "10.142.0.43:9100" 22:40:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:40:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:40:06 DEBUG }, 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:06 DEBUG }, 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG { 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0.00036666666666652263" 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "5171" 22:40:06 DEBUG "1.0357727332323379" 22:40:06 DEBUG ] 22:40:06 DEBUG "data": { 22:40:06 DEBUG ] 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG "result": [ 22:40:06 DEBUG ] 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG { 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG "metric": { 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG "pod": "ds-idrepo-2" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "ds-cts-2" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:40:06 DEBUG "pod": "ds-idrepo-2" 22:40:06 DEBUG }, 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:40:06 DEBUG "pod": "ds-idrepo-0" 22:40:06 DEBUG }, 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG { 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "data": { 22:40:06 DEBUG { 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "result": [ 22:40:06 DEBUG "data": { 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "result": [ 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:40:06 DEBUG { 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG { 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG "metric": { 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG }, 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:40:06 DEBUG "value": [ 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:40:06 DEBUG "instance": "10.142.0.107:9100" 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG }, 22:40:06 DEBUG 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG }, 22:40:06 DEBUG }, 22:40:06 DEBUG { 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG { 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "data": { 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG "data": { 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG { 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "result": [ 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG "result": [ 22:40:06 DEBUG [...] 22:40:06 DEBUG [...] 22:40:06 DEBUG { 22:40:06 DEBUG "data": { 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG { 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG 22:40:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:40:06 DEBUG "metric": { 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "ds-idrepo-2" 22:40:06 DEBUG "result": [ 22:40:06 DEBUG 22:40:06 DEBUG }, 22:40:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG }, 22:40:06 DEBUG { 22:40:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:40:06 DEBUG "value": [ 22:40:06 DEBUG }, 22:40:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:40:06 DEBUG "instance": "10.142.0.43:9100" 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "metric": { 22:40:06 DEBUG }, 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG }, 22:40:06 DEBUG }, 22:40:06 DEBUG 22:40:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "344.6" 22:40:06 DEBUG 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG }, 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG ] 22:40:06 DEBUG 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG 22:40:06 DEBUG "value": [ 22:40:06 DEBUG "0" 22:40:06 DEBUG } 22:40:06 DEBUG 22:40:06 DEBUG "0.08" 22:40:06 DEBUG "12159.599999999997" 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG ] 22:40:06 DEBUG ] 22:40:06 DEBUG [...] 22:40:06 DEBUG ] 22:40:06 DEBUG ] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG 22:40:06 DEBUG ] 22:40:06 DEBUG ] 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG } 22:40:06 DEBUG 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG "pod": "ds-idrepo-0" 22:40:06 DEBUG 22:40:06 DEBUG } 22:40:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG }, 22:40:06 DEBUG 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG }, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG "value": [ 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG "1721491456" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "am-7d597cf7d6-8ctft" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "am-7d597cf7d6-9wz9h" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "instance": "10.142.0.107:9100" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "instance": "10.142.0.43:9100" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "3192.1666666666665" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-cts-1" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-idrepo-2" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "ds-idrepo-0" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-cts-1" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-idrepo-2" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "ds-idrepo-0" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-idrepo-2" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "ds-idrepo-0" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-idrepo-2" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "ds-idrepo-0" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-cts-1" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0.00002182369618250684" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "pod": "ds-cts-1" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "instance": "10.142.0.107:9100" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "instance": "10.142.0.43:9100" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "0" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713217206" 22:40:06 INFO [http_cmd]: http status code OK 22:40:06 DEBUG --- status code --- 22:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:06 DEBUG --- http response --- 22:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:06 DEBUG ----- output ----- 22:40:06 DEBUG { 22:40:06 DEBUG "status": "success", 22:40:06 DEBUG "data": { 22:40:06 DEBUG "resultType": "vector", 22:40:06 DEBUG "result": [ 22:40:06 DEBUG { 22:40:06 DEBUG "metric": { 22:40:06 DEBUG "instance": "10.142.0.107:9100" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG [...] 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG 22:40:06 DEBUG "instance": "10.142.0.43:9100" 22:40:06 DEBUG }, 22:40:06 DEBUG "value": [ 22:40:06 DEBUG 1713217206, 22:40:06 DEBUG "13879923712" 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG ] 22:40:06 DEBUG } 22:40:06 DEBUG } 22:40:07 INFO 22:40:07 INFO [loop_until]: kubectl --namespace=xlou top pods 22:40:07 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:40:07 INFO [loop_until]: OK (rc = 0) 22:40:07 DEBUG --- stdout --- 22:40:07 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 28m 8599Mi am-7d597cf7d6-8vr8f 152m 9019Mi am-7d597cf7d6-9wz9h 587m 8923Mi ds-cts-0 1155m 8454Mi ds-cts-1 564m 8544Mi ds-cts-2 434m 8691Mi ds-idrepo-0 56m 4975Mi ds-idrepo-1 14m 2682Mi ds-idrepo-2 32m 2697Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 4m 1224Mi lodemon-58b6f5b7dd-dq2sv 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 541m 1328Mi 22:40:07 DEBUG --- stderr --- 22:40:07 DEBUG 22:40:08 INFO 22:40:08 INFO [loop_until]: kubectl --namespace=xlou top node 22:40:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:40:08 INFO [loop_until]: OK (rc = 0) 22:40:08 DEBUG --- stdout --- 22:40:08 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5070Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 225m 0% 10804Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 123m 0% 10718Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3096Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 68m 0% 5556Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 511m 1% 12802Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2989Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 226m 0% 10515Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 67m 0% 4898Mi 4% gke-xlou-cdm-ds-0f601751-j488 219m 0% 10342Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2434Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 433m 1% 10604Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 83m 0% 7167Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 527m 3% 5227Mi 8% 22:40:08 DEBUG --- stderr --- 22:40:08 DEBUG 22:40:34 INFO Finished: True 22:40:34 INFO Waiting for threads to register finish flag 22:41:08 INFO Done. Have a nice day! :) 127.0.0.1 - - [15/Apr/2024 22:41:08] "GET /monitoring/stop HTTP/1.1" 200 - 22:41:11 WARNING File /tmp/lodemon_data-20240415_215803/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [15/Apr/2024 22:41:15] "GET /monitoring/process HTTP/1.1" 200 -