==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-596c8449d-bbnvt Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-174e6abc-k8cj/10.142.0.65 Start Time: Thu, 25 Apr 2024 17:25:45 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=596c8449d skaffold.dev/run-id=7cdbbd59-cec2-4171-ad41-ffd8aa37f04e Annotations: Status: Running IP: 10.106.44.143 IPs: IP: 10.106.44.143 Controlled By: ReplicaSet/lodemon-596c8449d Containers: lodemon: Container ID: containerd://d181f8d0aa7a0140dfd82fb720b4d6820e8930289ee02642bc1973aa3ef1a688 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:8f3ffb07d87daf5d0bb82f4e1d6e9126b62ec155fd32b51512c2619dd67dd55c Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Thu, 25 Apr 2024 17:25:46 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-55tkg (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-55tkg: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 18m default-scheduler Successfully assigned xlou/lodemon-596c8449d-bbnvt to gke-xlou-cdm-ds-174e6abc-k8cj Normal Pulling 18m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 18m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 220.095003ms (220.103307ms including waiting) Normal Created 18m kubelet Created container lodemon Normal Started 18m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.44.143:8080 Press CTRL+C to quit 18:26:21 INFO 18:26:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:21 INFO [loop_until]: OK (rc = 0) 18:26:21 DEBUG --- stdout --- 18:26:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:21 DEBUG --- stderr --- 18:26:21 DEBUG 18:26:21 INFO 18:26:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:21 INFO [loop_until]: OK (rc = 0) 18:26:21 DEBUG --- stdout --- 18:26:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:21 DEBUG --- stderr --- 18:26:21 DEBUG 18:26:21 INFO 18:26:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:21 INFO [loop_until]: OK (rc = 0) 18:26:21 DEBUG --- stdout --- 18:26:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:21 DEBUG --- stderr --- 18:26:21 DEBUG 18:26:21 INFO 18:26:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:22 INFO [loop_until]: OK (rc = 0) 18:26:22 DEBUG --- stdout --- 18:26:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:22 DEBUG --- stderr --- 18:26:22 DEBUG 18:26:22 INFO 18:26:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:22 INFO [loop_until]: OK (rc = 0) 18:26:22 DEBUG --- stdout --- 18:26:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:22 DEBUG --- stderr --- 18:26:22 DEBUG 18:26:22 INFO 18:26:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:22 INFO [loop_until]: OK (rc = 0) 18:26:22 DEBUG --- stdout --- 18:26:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:22 DEBUG --- stderr --- 18:26:22 DEBUG 18:26:22 INFO 18:26:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:22 INFO [loop_until]: OK (rc = 0) 18:26:22 DEBUG --- stdout --- 18:26:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:22 DEBUG --- stderr --- 18:26:22 DEBUG 18:26:22 INFO 18:26:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:22 INFO [loop_until]: OK (rc = 0) 18:26:22 DEBUG --- stdout --- 18:26:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:22 DEBUG --- stderr --- 18:26:22 DEBUG 18:26:22 INFO 18:26:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:22 INFO [loop_until]: OK (rc = 0) 18:26:22 DEBUG --- stdout --- 18:26:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:22 DEBUG --- stderr --- 18:26:22 DEBUG 18:26:22 INFO 18:26:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:22 INFO [loop_until]: OK (rc = 0) 18:26:22 DEBUG --- stdout --- 18:26:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:22 DEBUG --- stderr --- 18:26:22 DEBUG 18:26:22 INFO 18:26:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:23 INFO [loop_until]: OK (rc = 0) 18:26:23 DEBUG --- stdout --- 18:26:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:23 DEBUG --- stderr --- 18:26:23 DEBUG 18:26:23 INFO 18:26:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:23 INFO [loop_until]: OK (rc = 0) 18:26:23 DEBUG --- stdout --- 18:26:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:23 DEBUG --- stderr --- 18:26:23 DEBUG 18:26:23 INFO 18:26:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:23 INFO [loop_until]: OK (rc = 0) 18:26:23 DEBUG --- stdout --- 18:26:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:23 DEBUG --- stderr --- 18:26:23 DEBUG 18:26:23 INFO 18:26:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:23 INFO [loop_until]: OK (rc = 0) 18:26:23 DEBUG --- stdout --- 18:26:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:23 DEBUG --- stderr --- 18:26:23 DEBUG 18:26:23 INFO 18:26:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:23 INFO [loop_until]: OK (rc = 0) 18:26:23 DEBUG --- stdout --- 18:26:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:23 DEBUG --- stderr --- 18:26:23 DEBUG 18:26:23 INFO 18:26:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:23 INFO [loop_until]: OK (rc = 0) 18:26:23 DEBUG --- stdout --- 18:26:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:23 DEBUG --- stderr --- 18:26:23 DEBUG 18:26:23 INFO 18:26:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:23 INFO [loop_until]: OK (rc = 0) 18:26:23 DEBUG --- stdout --- 18:26:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:23 DEBUG --- stderr --- 18:26:23 DEBUG 18:26:23 INFO 18:26:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:24 INFO [loop_until]: OK (rc = 0) 18:26:24 DEBUG --- stdout --- 18:26:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:24 DEBUG --- stderr --- 18:26:24 DEBUG 18:26:24 INFO 18:26:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:24 INFO [loop_until]: OK (rc = 0) 18:26:24 DEBUG --- stdout --- 18:26:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:24 DEBUG --- stderr --- 18:26:24 DEBUG 18:26:24 INFO 18:26:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:24 INFO [loop_until]: OK (rc = 0) 18:26:24 DEBUG --- stdout --- 18:26:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:24 DEBUG --- stderr --- 18:26:24 DEBUG 18:26:24 INFO 18:26:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:24 INFO [loop_until]: OK (rc = 0) 18:26:24 DEBUG --- stdout --- 18:26:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:24 DEBUG --- stderr --- 18:26:24 DEBUG 18:26:24 INFO 18:26:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:24 INFO [loop_until]: OK (rc = 0) 18:26:24 DEBUG --- stdout --- 18:26:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:24 DEBUG --- stderr --- 18:26:24 DEBUG 18:26:24 INFO 18:26:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:24 INFO [loop_until]: OK (rc = 0) 18:26:24 DEBUG --- stdout --- 18:26:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:24 DEBUG --- stderr --- 18:26:24 DEBUG 18:26:24 INFO 18:26:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:24 INFO [loop_until]: OK (rc = 0) 18:26:24 DEBUG --- stdout --- 18:26:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:24 DEBUG --- stderr --- 18:26:24 DEBUG 18:26:24 INFO 18:26:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:25 INFO [loop_until]: OK (rc = 0) 18:26:25 DEBUG --- stdout --- 18:26:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:25 DEBUG --- stderr --- 18:26:25 DEBUG 18:26:25 INFO 18:26:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:25 INFO [loop_until]: OK (rc = 0) 18:26:25 DEBUG --- stdout --- 18:26:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:25 DEBUG --- stderr --- 18:26:25 DEBUG 18:26:25 INFO 18:26:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:26:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:25 INFO [loop_until]: OK (rc = 0) 18:26:25 DEBUG --- stdout --- 18:26:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:26:25 DEBUG --- stderr --- 18:26:25 DEBUG 18:26:25 INFO Initializing monitoring instance threads 18:26:25 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 18:26:25 INFO Starting instance threads 18:26:25 INFO 18:26:25 INFO Thread started 18:26:25 INFO [loop_until]: kubectl --namespace=xlou top node 18:26:25 INFO 18:26:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:25 INFO Thread started 18:26:25 INFO [loop_until]: kubectl --namespace=xlou top pods 18:26:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO Thread started 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG --- status code --- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG --- status code --- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG --- status code --- 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 DEBUG --- status code --- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- status code --- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG --- status code --- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 DEBUG --- http response --- 18:26:25 INFO Thread started 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO [loop_until]: OK (rc = 0) 18:26:25 DEBUG --- http response --- 18:26:25 INFO Thread started 18:26:25 DEBUG --- http response --- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG --- stdout --- 18:26:25 INFO [loop_until]: OK (rc = 0) 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG { 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG { 18:26:25 DEBUG ----- output ----- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 DEBUG { 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG { 18:26:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 9m 3282Mi am-769b98f8d5-phmxx 10m 2700Mi am-769b98f8d5-zbtk2 8m 3277Mi ds-cts-0 19m 483Mi ds-cts-1 16m 462Mi ds-cts-2 18m 468Mi ds-idrepo-0 27m 671Mi ds-idrepo-1 28m 683Mi ds-idrepo-2 23m 667Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 19m 1091Mi idm-6c95d7588f-n2wtz 10m 1058Mi lodemon-596c8449d-bbnvt 54m 54Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 1m 15Mi 18:26:25 DEBUG --- stdout --- 18:26:25 DEBUG ----- output ----- 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 DEBUG ----- output ----- 18:26:25 INFO Thread started 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG { 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG --- stderr --- 18:26:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 87m 0% 2891Mi 4% gke-xlou-cdm-default-pool-9d1e5395-4r9x 75m 0% 1852Mi 3% gke-xlou-cdm-default-pool-9d1e5395-jnd9 73m 0% 1896Mi 3% gke-xlou-cdm-default-pool-9d1e5395-qd5b 73m 0% 1793Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 66m 0% 5886Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 75m 0% 1675Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 76m 0% 1632Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 67m 0% 4902Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 78m 0% 2954Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 229m 1% 3640Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 71m 0% 4258Mi 7% gke-xlou-cdm-ds-174e6abc-mm5l 68m 0% 1406Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 79m 0% 2689Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 69m 0% 3596Mi 6% 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG --- http response --- 18:26:25 INFO Thread started 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "data": { 18:26:25 DEBUG { 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG "data": { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG { 18:26:25 DEBUG "data": { 18:26:25 DEBUG { 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG "data": { 18:26:25 DEBUG 18:26:25 DEBUG --- stderr --- 18:26:25 DEBUG "status": "success", 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 DEBUG "data": { 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG "data": { 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "data": { 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG { 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG 18:26:25 DEBUG "data": { 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG ----- output ----- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 DEBUG "result": [ 18:26:25 INFO Thread started 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG "data": { 18:26:25 DEBUG "data": { 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "result": [ 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 DEBUG "data": { 18:26:25 DEBUG "result": [ 18:26:25 DEBUG "data": { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG "result": [ 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "result": [ 18:26:25 DEBUG { 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 18:26:25 DEBUG "result": [ 18:26:25 DEBUG { 18:26:25 DEBUG "result": [ 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG { 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "data": { 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG { 18:26:25 DEBUG "result": [ 18:26:25 DEBUG { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "status": "success", 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714065985" 18:26:25 INFO Thread started 18:26:25 DEBUG "result": [ 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG "result": [ 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG { 18:26:25 DEBUG "metric": { 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 DEBUG "result": [ 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "result": [ 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "metric": { 18:26:25 DEBUG { 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "data": { 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:26:25 DEBUG "data": { 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714065985" 18:26:25 DEBUG { 18:26:25 INFO Thread started 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG { 18:26:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:26:25 DEBUG { 18:26:25 DEBUG "result": [ 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 DEBUG "instance": "10.142.0.24:9100" 18:26:25 DEBUG "metric": { 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG "instance": "10.142.0.24:9100" 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:26:25 DEBUG }, 18:26:25 DEBUG "resultType": "vector", 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG "metric": { 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714065985" 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 INFO Thread started 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:26:25 DEBUG }, 18:26:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:26:25 DEBUG "data": { 18:26:25 DEBUG "metric": { 18:26:25 DEBUG }, 18:26:25 DEBUG "metric": { 18:26:25 DEBUG { 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG }, 18:26:25 DEBUG "instance": "10.142.0.24:9100" 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG }, 18:26:25 DEBUG "result": [ 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "result": [ 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:26:25 DEBUG --- http response --- 18:26:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714065985" 18:26:25 DEBUG --- http response --- 18:26:25 INFO Thread started 18:26:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:26:25 DEBUG "data": { 18:26:25 DEBUG }, 18:26:25 DEBUG "value": [ 18:26:25 DEBUG }, 18:26:25 DEBUG "resultType": "vector", 18:26:25 ERROR [http_cmd]: ERROR 18:26:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "pod": "ds-cts-0" 18:26:25 DEBUG "metric": { 18:26:25 DEBUG { 18:26:25 DEBUG "value": [ 18:26:25 DEBUG }, 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "value": [ 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG }, 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 INFO All threads has been started 127.0.0.1 - - [25/Apr/2024 18:26:25] "GET /monitoring/start HTTP/1.1" 200 - 18:26:25 DEBUG }, 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "result": [ 18:26:25 ERROR --- status code --- 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG "pod": "ds-idrepo-1" 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:26:25 DEBUG 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG 18:26:25 DEBUG "metric": { 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "value": [ 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG "value": [ 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG "result": [ 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG }, 18:26:25 DEBUG "data": { 18:26:25 DEBUG 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG 18:26:25 DEBUG "pod": "ds-idrepo-1" 18:26:25 DEBUG "data": { 18:26:25 DEBUG 18:26:25 DEBUG "pod": "ds-idrepo-1" 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG --- status code --- 18:26:25 INFO [http_cmd]: http status code OK 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "metric": { 18:26:25 ERROR --- http response --- 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG [...] 18:26:25 DEBUG }, 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- status code --- 18:26:25 DEBUG "metric": { 18:26:25 DEBUG 18:26:25 DEBUG [...] 18:26:25 DEBUG 18:26:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:26:25 DEBUG 18:26:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:26:25 DEBUG [...] 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "result": [ 18:26:25 DEBUG [...] 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG [...] 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "result": [ 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG "data": { 18:26:25 DEBUG "data": { 18:26:25 DEBUG 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:26:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:26:25 DEBUG [...] 18:26:25 DEBUG 18:26:25 DEBUG [...] 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG [...] 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:26:25 DEBUG "data": { 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG [...] 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG [...] 18:26:25 DEBUG --- http response --- 18:26:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG [...] 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "metric": { 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "metric": { 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "data": { 18:26:25 DEBUG 18:26:25 DEBUG "result": [ 18:26:25 DEBUG "result": [ 18:26:25 DEBUG 18:26:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG [...] 18:26:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:26:25 DEBUG [...] 18:26:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG "result": [ 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "pod": "ds-cts-0" 18:26:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:26:25 DEBUG 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG ----- output ----- 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG "instance": "10.142.0.74:9100" 18:26:25 DEBUG 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG { 18:26:25 DEBUG "instance": "10.142.0.74:9100" 18:26:25 DEBUG [...] 18:26:25 DEBUG }, 18:26:25 DEBUG }, 18:26:25 DEBUG [...] 18:26:25 DEBUG "result": [ 18:26:25 DEBUG 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "metric": { 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:26:25 DEBUG }, 18:26:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG }, 18:26:25 DEBUG "instance": "10.142.0.74:9100" 18:26:25 DEBUG "data": { 18:26:25 DEBUG "metric": { 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG { 18:26:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:26:25 DEBUG "pod": "ds-cts-0" 18:26:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:26:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:26:25 DEBUG "status": "success", 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG "value": [ 18:26:25 DEBUG }, 18:26:25 DEBUG [...] 18:26:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:26:25 DEBUG 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG }, 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG "metric": { 18:26:25 DEBUG }, 18:26:25 DEBUG }, 18:26:25 DEBUG }, 18:26:25 DEBUG }, 18:26:25 DEBUG "data": { 18:26:25 DEBUG [...] 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG "pod": "ds-cts-2" 18:26:25 DEBUG "0" 18:26:25 DEBUG "pod": "ds-idrepo-0" 18:26:25 DEBUG 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "result": [ 18:26:25 DEBUG }, 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "0" 18:26:25 DEBUG 18:26:25 DEBUG "pod": "ds-idrepo-1" 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "resultType": "vector", 18:26:25 DEBUG 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG "0.03560579552816596" 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG }, 18:26:25 DEBUG ] 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 DEBUG "0" 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG { 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "4010.22992335888" 18:26:25 DEBUG "pod": "ds-idrepo-0" 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG "pod": "ds-idrepo-0" 18:26:25 DEBUG }, 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG "result": [ 18:26:25 DEBUG 18:26:25 DEBUG "0" 18:26:25 DEBUG ] 18:26:25 DEBUG "0.0003289416521212558" 18:26:25 DEBUG 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG "value": [ 18:26:25 DEBUG } 18:26:25 DEBUG "value": [ 18:26:25 DEBUG [...] 18:26:25 DEBUG ] 18:26:25 DEBUG "8612.06264578474" 18:26:25 DEBUG "metric": { 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG }, 18:26:25 DEBUG [...] 18:26:25 DEBUG } 18:26:25 DEBUG }, 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "0" 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG "115384320" 18:26:25 DEBUG { 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:26:25 DEBUG "0" 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG ] 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG "instance": "10.142.0.24:9100" 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG "metric": { 18:26:25 DEBUG "pod": "am-769b98f8d5-4ftrv" 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG }, 18:26:25 DEBUG ] 18:26:25 DEBUG "0" 18:26:25 DEBUG } 18:26:25 DEBUG "0" 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG [...] 18:26:25 DEBUG [...] 18:26:25 DEBUG } 18:26:25 DEBUG "instance": "10.142.0.24:9100" 18:26:25 DEBUG }, 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG "value": [ 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG "value": [ 18:26:25 DEBUG [...] 18:26:25 DEBUG } 18:26:25 DEBUG "0" 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG "0" 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG }, 18:26:25 DEBUG "value": [ 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG "pod": "ds-cts-2" 18:26:25 DEBUG ] 18:26:25 DEBUG [...] 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG "0" 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG ] 18:26:25 DEBUG }, 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG }, 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG "0" 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG "value": [ 18:26:25 DEBUG 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG "value": [ 18:26:25 DEBUG ] 18:26:25 DEBUG 18:26:25 DEBUG "pod": "ds-cts-2" 18:26:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:26:25 DEBUG 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG [...] 18:26:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:26:25 DEBUG } 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG }, 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG "0" 18:26:25 DEBUG 18:26:25 DEBUG }, 18:26:25 DEBUG } 18:26:25 DEBUG "0.03170611111111111" 18:26:25 DEBUG } 18:26:25 DEBUG "pod": "ds-cts-2" 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "value": [ 18:26:25 DEBUG [...] 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG 18:26:25 DEBUG "value": [ 18:26:25 DEBUG ] 18:26:25 DEBUG }, 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG } 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "0" 18:26:25 DEBUG "1" 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG "instance": "10.142.0.74:9100" 18:26:25 DEBUG "0" 18:26:25 DEBUG ] 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG ] 18:26:25 DEBUG ] 18:26:25 DEBUG 18:26:25 DEBUG } 18:26:25 DEBUG }, 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG "0" 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG "instance": "10.142.0.74:9100" 18:26:25 DEBUG } 18:26:25 DEBUG "value": [ 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG ] 18:26:25 DEBUG ] 18:26:25 DEBUG }, 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG "value": [ 18:26:25 DEBUG "4515484672" 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG 1714065985, 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG "0.0001332889036986932" 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG ] 18:26:25 DEBUG } 18:26:25 DEBUG } 18:26:25 DEBUG } 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO 18:27:25 INFO [loop_until]: kubectl --namespace=xlou top pods 18:27:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:27:25 INFO 18:27:25 INFO [loop_until]: kubectl --namespace=xlou top node 18:27:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066045" 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 DEBUG --- status code --- 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066045" 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG --- status code --- 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG --- http response --- 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG --- http response --- 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG { 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG { 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG { 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG "data": { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG "data": { 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066045" 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG { 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "metric": { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "result": [ 18:27:25 DEBUG { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:27:25 DEBUG "data": { 18:27:25 DEBUG }, 18:27:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:27:25 DEBUG "pod": "ds-idrepo-1" 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "data": { 18:27:25 DEBUG }, 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "value": [ 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG { 18:27:25 DEBUG "result": [ 18:27:25 DEBUG 18:27:25 DEBUG { 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "metric": { 18:27:25 DEBUG { 18:27:25 DEBUG 18:27:25 DEBUG "metric": { 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "instance": "10.142.0.24:9100" 18:27:25 DEBUG "metric": { 18:27:25 DEBUG 18:27:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:27:25 DEBUG [...] 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG "value": [ 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG "pod": "ds-idrepo-0" 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG [...] 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG "0" 18:27:25 DEBUG }, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 18:27:25 DEBUG ] 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "0.5261794122006451" 18:27:25 DEBUG 18:27:25 DEBUG } 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG "0" 18:27:25 DEBUG "instance": "10.142.0.74:9100" 18:27:25 DEBUG 18:27:25 DEBUG ] 18:27:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:27:25 DEBUG ] 18:27:25 DEBUG "8008414.593588712" 18:27:25 DEBUG ] 18:27:25 DEBUG }, 18:27:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:27:25 DEBUG } 18:27:25 DEBUG }, 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG "value": [ 18:27:25 DEBUG }, 18:27:25 DEBUG ] 18:27:25 DEBUG "value": [ 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG } 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG "11365.733333333334" 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG } 18:27:25 DEBUG "0.3852272621183879" 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG "0" 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 ERROR [http_cmd]: ERROR 18:27:25 ERROR --- status code --- 18:27:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:27:25 ERROR --- http response --- 18:27:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:27:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [loop_until]: OK (rc = 0) 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066045" 18:27:25 DEBUG --- stdout --- 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 9m 3283Mi am-769b98f8d5-phmxx 126m 2900Mi am-769b98f8d5-zbtk2 143m 3319Mi ds-cts-0 10m 497Mi ds-cts-1 138m 479Mi ds-cts-2 19m 480Mi ds-idrepo-0 1361m 1339Mi ds-idrepo-1 1438m 1296Mi ds-idrepo-2 261m 671Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 9m 1092Mi idm-6c95d7588f-n2wtz 16m 1059Mi lodemon-596c8449d-bbnvt 2m 61Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 876m 196Mi 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066045" 18:27:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066045" 18:27:25 DEBUG --- stderr --- 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [loop_until]: OK (rc = 0) 18:27:25 DEBUG --- status code --- 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG --- stdout --- 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 1019m 6% 3162Mi 5% gke-xlou-cdm-default-pool-9d1e5395-4r9x 485m 3% 1947Mi 3% gke-xlou-cdm-default-pool-9d1e5395-jnd9 1282m 8% 2380Mi 4% gke-xlou-cdm-default-pool-9d1e5395-qd5b 63m 0% 1802Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 209m 1% 5929Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 68m 0% 1661Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 166m 1% 1621Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 228m 1% 4923Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 80m 0% 2935Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 162m 1% 3665Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 189m 1% 4456Mi 7% gke-xlou-cdm-ds-174e6abc-mm5l 64m 0% 1403Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 63m 0% 2690Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 959m 6% 3739Mi 6% 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG --- stderr --- 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:27:25 INFO [http_cmd]: http status code OK 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:27:25 DEBUG 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG --- status code --- 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:27:25 DEBUG { 18:27:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG { 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG --- http response --- 18:27:25 DEBUG { 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG "data": { 18:27:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "data": { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG ----- output ----- 18:27:25 DEBUG "data": { 18:27:25 DEBUG { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "data": { 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "data": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG "result": [ 18:27:25 DEBUG { 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG { 18:27:25 DEBUG "status": "success", 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "data": { 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG "result": [ 18:27:25 DEBUG { 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "data": { 18:27:25 DEBUG { 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG { 18:27:25 DEBUG { 18:27:25 DEBUG "instance": "10.142.0.24:9100" 18:27:25 DEBUG "resultType": "vector", 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:27:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:27:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:27:25 DEBUG "instance": "10.142.0.24:9100" 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "metric": { 18:27:25 DEBUG }, 18:27:25 DEBUG "result": [ 18:27:25 DEBUG "pod": "ds-cts-0" 18:27:25 DEBUG { 18:27:25 DEBUG "pod": "ds-idrepo-1" 18:27:25 DEBUG "pod": "ds-idrepo-1" 18:27:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:27:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:27:25 DEBUG "pod": "ds-cts-0" 18:27:25 DEBUG "pod": "ds-cts-0" 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG "pod": "ds-idrepo-1" 18:27:25 DEBUG }, 18:27:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:27:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:27:25 DEBUG "value": [ 18:27:25 DEBUG { 18:27:25 DEBUG }, 18:27:25 DEBUG "metric": { 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG }, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG "metric": { 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "instance": "10.142.0.24:9100" 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG "instance": "10.142.0.24:9100" 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "instance": "10.142.0.74:9100" 18:27:25 DEBUG [...] 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG "pod": "login-ui-5fdf49fc66-rzjdb" 18:27:25 DEBUG "pod": "login-ui-5fdf49fc66-rzjdb" 18:27:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:27:25 DEBUG "instance": "10.142.0.74:9100" 18:27:25 DEBUG 18:27:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:27:25 DEBUG 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG "pod": "ds-cts-2" 18:27:25 DEBUG 18:27:25 DEBUG "pod": "ds-idrepo-0" 18:27:25 DEBUG "pod": "ds-idrepo-0" 18:27:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:27:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:27:25 DEBUG "pod": "ds-cts-2" 18:27:25 DEBUG "pod": "ds-cts-2" 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG "pod": "ds-cts-2" 18:27:25 DEBUG }, 18:27:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:27:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG }, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG }, 18:27:25 DEBUG }, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "instance": "10.142.0.74:9100" 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "5350.133333333333" 18:27:25 DEBUG "instance": "10.142.0.74:9100" 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG }, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG "361340928" 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG ] 18:27:25 DEBUG }, 18:27:25 DEBUG "0" 18:27:25 DEBUG "value": [ 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG "0" 18:27:25 DEBUG ] 18:27:25 DEBUG "0" 18:27:25 DEBUG "0" 18:27:25 DEBUG } 18:27:25 DEBUG "value": [ 18:27:25 DEBUG ] 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG 1714066045, 18:27:25 DEBUG } 18:27:25 DEBUG "4515714048" 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG "0.00013333333333325942" 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG ] 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:27:25 DEBUG } 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 ERROR [http_cmd]: ERROR 18:28:25 ERROR --- status code --- 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:28:25 DEBUG --- status code --- 18:28:25 ERROR --- http response --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:28:25 DEBUG --- http response --- 18:28:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "data": { 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "result": [ 18:28:25 DEBUG { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "instance": "10.142.0.24:9100" 18:28:25 DEBUG }, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG "instance": "10.142.0.74:9100" 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG }, 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG "value": [ 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG "10271.3" 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG ] 18:28:25 DEBUG { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG } 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG ] 18:28:25 DEBUG "data": { 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG } 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "result": [ 18:28:25 DEBUG } 18:28:25 DEBUG "result": [ 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG "metric": { 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "pod": "ds-idrepo-1" 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:28:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:28:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:28:25 DEBUG }, 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG { 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG [...] 18:28:25 DEBUG [...] 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "data": { 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "data": { 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "result": [ 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "pod": "ds-idrepo-0" 18:28:25 DEBUG "result": [ 18:28:25 DEBUG { 18:28:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:28:25 DEBUG "pod": "am-769b98f8d5-4ftrv" 18:28:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:28:25 DEBUG }, 18:28:25 DEBUG { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:28:25 DEBUG }, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "0" 18:28:25 DEBUG }, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "0" 18:28:25 DEBUG "1358567.9381700323" 18:28:25 DEBUG "0.25337363212055763" 18:28:25 DEBUG ] 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:28:25 DEBUG "pod": "am-769b98f8d5-4ftrv" 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "0" 18:28:25 DEBUG "0.16771712166541983" 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066105" 18:28:25 INFO 18:28:25 INFO [loop_until]: kubectl --namespace=xlou top pods 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066105" 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG --- status code --- 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG { 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG "data": { 18:28:25 DEBUG { 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "data": { 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 INFO 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG { 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG ----- output ----- 18:28:25 INFO [loop_until]: kubectl --namespace=xlou top node 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "data": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG { 18:28:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "pod": "ds-cts-0" 18:28:25 DEBUG { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG }, 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG { 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "instance": "10.142.0.24:9100" 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:28:25 DEBUG "pod": "ds-idrepo-1" 18:28:25 DEBUG "metric": { 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG "result": [ 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG { 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "metric": { 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG "instance": "10.142.0.24:9100" 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG "value": [ 18:28:25 DEBUG [...] 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG "pod": "ds-cts-2" 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "instance": "10.142.0.74:9100" 18:28:25 DEBUG [...] 18:28:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:28:25 DEBUG "pod": "ds-idrepo-0" 18:28:25 DEBUG 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:28:25 DEBUG "0" 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG }, 18:28:25 DEBUG ] 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG } 18:28:25 DEBUG "4515953664" 18:28:25 DEBUG "instance": "10.142.0.74:9100" 18:28:25 DEBUG "0" 18:28:25 DEBUG "0" 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG }, 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG "0" 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG "value": [ 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG "0" 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066105" 18:28:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066105" 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [loop_until]: OK (rc = 0) 18:28:25 DEBUG --- status code --- 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- stdout --- 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 162m 3368Mi am-769b98f8d5-phmxx 138m 3621Mi am-769b98f8d5-zbtk2 139m 3356Mi ds-cts-0 10m 497Mi ds-cts-1 10m 480Mi ds-cts-2 10m 483Mi ds-idrepo-0 4208m 1899Mi ds-idrepo-1 3744m 1742Mi ds-idrepo-2 3665m 1842Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 5m 1093Mi idm-6c95d7588f-n2wtz 8m 1059Mi lodemon-596c8449d-bbnvt 2m 61Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 231m 240Mi 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- stderr --- 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 INFO [http_cmd]: http status code OK 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG --- status code --- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:25 DEBUG { 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:28:25 DEBUG "data": { 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG --- http response --- 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:25 DEBUG "data": { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG ----- output ----- 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG { 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "result": [ 18:28:25 DEBUG { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "data": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "result": [ 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG "status": "success", 18:28:25 DEBUG { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "pod": "ds-idrepo-1" 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "data": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG "metric": { 18:28:25 DEBUG { 18:28:25 DEBUG "result": [ 18:28:25 DEBUG }, 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "metric": { 18:28:25 DEBUG "pod": "ds-idrepo-1" 18:28:25 DEBUG "instance": "10.142.0.24:9100" 18:28:25 DEBUG "resultType": "vector", 18:28:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:28:25 DEBUG { 18:28:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:28:25 DEBUG "metric": { 18:28:25 DEBUG { 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:28:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:28:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG "result": [ 18:28:25 DEBUG }, 18:28:25 DEBUG "metric": { 18:28:25 DEBUG }, 18:28:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:28:25 DEBUG "metric": { 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG { 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "pod": "ds-cts-0" 18:28:25 DEBUG "value": [ 18:28:25 DEBUG }, 18:28:25 DEBUG "pod": "ds-cts-0" 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG "metric": { 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "instance": "10.142.0.24:9100" 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 INFO [loop_until]: OK (rc = 0) 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG --- stdout --- 18:28:25 DEBUG "value": [ 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3725m 23% 4058Mi 6% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3848m 24% 3144Mi 5% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4175m 26% 3101Mi 5% gke-xlou-cdm-default-pool-9d1e5395-qd5b 63m 0% 1813Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 200m 1% 5999Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 67m 0% 1683Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 66m 0% 1628Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 209m 1% 5003Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 73m 0% 2942Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 174m 1% 3651Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 186m 1% 5179Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 67m 0% 1395Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 66m 0% 2684Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 303m 1% 3784Mi 6% 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG --- stderr --- 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG "pod": "ds-idrepo-0" 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "instance": "10.142.0.74:9100" 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:28:25 DEBUG 18:28:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:28:25 DEBUG 18:28:25 DEBUG 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "pod": "am-769b98f8d5-4ftrv" 18:28:25 DEBUG "pod": "ds-cts-2" 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG [...] 18:28:25 DEBUG "value": [ 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG }, 18:28:25 DEBUG "pod": "am-769b98f8d5-4ftrv" 18:28:25 DEBUG 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG }, 18:28:25 DEBUG }, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:28:25 DEBUG 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "pod": "ds-cts-2" 18:28:25 DEBUG "value": [ 18:28:25 DEBUG }, 18:28:25 DEBUG "pod": "ds-cts-2" 18:28:25 DEBUG "0" 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "353869824" 18:28:25 DEBUG }, 18:28:25 DEBUG 18:28:25 DEBUG "4397.533333333333" 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG }, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG }, 18:28:25 DEBUG ] 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG ] 18:28:25 DEBUG "value": [ 18:28:25 DEBUG 18:28:25 DEBUG ] 18:28:25 DEBUG "0" 18:28:25 DEBUG "value": [ 18:28:25 DEBUG "0" 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "value": [ 18:28:25 DEBUG } 18:28:25 DEBUG "0" 18:28:25 DEBUG "0" 18:28:25 DEBUG } 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG "instance": "10.142.0.74:9100" 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG ] 18:28:25 DEBUG "893.542757417103" 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG "0" 18:28:25 DEBUG }, 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG "0" 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG "0" 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG "value": [ 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG 1714066105, 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG "0.00020000000000000757" 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG } 18:28:25 DEBUG ] 18:28:25 DEBUG } 18:28:25 DEBUG } 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066165" 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:29:25 ERROR [http_cmd]: ERROR 18:29:25 ERROR --- status code --- 18:29:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:29:25 ERROR --- http response --- 18:29:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:29:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "ds-idrepo-1" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "pod": "ds-idrepo-0" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "0" 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066165" 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "0.19778412993268893" 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "instance": "10.142.0.24:9100" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "instance": "10.142.0.74:9100" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "11254.9" 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG --- status code --- 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG { 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "data": { 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:29:25 DEBUG }, 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "data": { 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG "result": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "value": [ 18:29:25 DEBUG { 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "metric": { 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:29:25 DEBUG [...] 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "8066429.578588063" 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG "0" 18:29:25 DEBUG ] 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG "0.23803580531073118" 18:29:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG ] 18:29:25 DEBUG }, 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG "value": [ 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG "0" 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "ds-idrepo-1" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "pod": "ds-idrepo-0" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "0" 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066165" 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "ds-cts-0" 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG "instance": "10.142.0.24:9100" 18:29:25 DEBUG }, 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG { 18:29:25 DEBUG { 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG "data": { 18:29:25 DEBUG "data": { 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "result": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG { 18:29:25 DEBUG { 18:29:25 DEBUG 18:29:25 DEBUG "pod": "ds-cts-2" 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "instance": "10.142.0.74:9100" 18:29:25 DEBUG }, 18:29:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:29:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG }, 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "value": [ 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "0" 18:29:25 DEBUG 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG 18:29:25 DEBUG "4516295680" 18:29:25 DEBUG ] 18:29:25 DEBUG 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG [...] 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:25 DEBUG [...] 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG 18:29:25 DEBUG { 18:29:25 DEBUG 18:29:25 DEBUG } 18:29:25 DEBUG 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG 18:29:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:29:25 DEBUG "data": { 18:29:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:29:25 DEBUG }, 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG { 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "0" 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "0" 18:29:25 DEBUG ] 18:29:25 DEBUG "instance": "10.142.0.24:9100" 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG }, 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG "value": [ 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG 18:29:25 DEBUG } 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "instance": "10.142.0.74:9100" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "0" 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066165" 18:29:25 INFO 18:29:25 INFO [loop_until]: kubectl --namespace=xlou top pods 18:29:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:29:25 DEBUG "instance": "10.142.0.24:9100" 18:29:25 DEBUG }, 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:29:25 DEBUG "instance": "10.142.0.74:9100" 18:29:25 DEBUG }, 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "0" 18:29:25 DEBUG "0.0001666666666665151" 18:29:25 DEBUG ] 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 INFO [loop_until]: kubectl --namespace=xlou top node 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "ds-idrepo-1" 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 DEBUG 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066165" 18:29:25 DEBUG 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066165" 18:29:25 DEBUG 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG "pod": "ds-cts-2" 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG }, 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- status code --- 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG "value": [ 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG --- http response --- 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG "0" 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG ----- output ----- 18:29:25 INFO [http_cmd]: http status code OK 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG ] 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG { 18:29:25 DEBUG --- status code --- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG } 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:25 DEBUG { 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG { 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG ] 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:29:25 DEBUG "data": { 18:29:25 DEBUG --- http response --- 18:29:25 DEBUG { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:25 DEBUG { 18:29:25 DEBUG } 18:29:25 DEBUG { 18:29:25 DEBUG "data": { 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "data": { 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG } 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG { 18:29:25 DEBUG "result": [ 18:29:25 DEBUG ----- output ----- 18:29:25 DEBUG "data": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG { 18:29:25 DEBUG "data": { 18:29:25 DEBUG "data": { 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG { 18:29:25 DEBUG { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG { 18:29:25 DEBUG "data": { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "status": "success", 18:29:25 DEBUG "result": [ 18:29:25 DEBUG { 18:29:25 DEBUG "data": { 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:29:25 DEBUG "data": { 18:29:25 DEBUG { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG { 18:29:25 DEBUG { 18:29:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:29:25 DEBUG "result": [ 18:29:25 DEBUG }, 18:29:25 DEBUG "resultType": "vector", 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "pod": "ds-idrepo-1" 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "metric": { 18:29:25 DEBUG }, 18:29:25 DEBUG { 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "result": [ 18:29:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:29:25 DEBUG }, 18:29:25 DEBUG { 18:29:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:29:25 DEBUG "instance": "10.142.0.24:9100" 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "metric": { 18:29:25 DEBUG 18:29:25 DEBUG { 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "metric": { 18:29:25 DEBUG }, 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG "pod": "ds-cts-0" 18:29:25 DEBUG 18:29:25 DEBUG "metric": { 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG "pod": "ds-cts-0" 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "value": [ 18:29:25 DEBUG [...] 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG [...] 18:29:25 DEBUG 18:29:25 DEBUG "pod": "ds-idrepo-0" 18:29:25 DEBUG 18:29:25 DEBUG 18:29:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:29:25 DEBUG "instance": "10.142.0.74:9100" 18:29:25 DEBUG 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:29:25 DEBUG }, 18:29:25 DEBUG }, 18:29:25 DEBUG 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 18:29:25 DEBUG }, 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "pod": "ds-cts-2" 18:29:25 DEBUG "0" 18:29:25 DEBUG 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:29:25 DEBUG "value": [ 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG }, 18:29:25 DEBUG ] 18:29:25 DEBUG "pod": "ds-cts-2" 18:29:25 DEBUG 1714066165, 18:29:25 INFO [loop_until]: OK (rc = 0) 18:29:25 DEBUG "0" 18:29:25 DEBUG }, 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG "350679040" 18:29:25 DEBUG "5219" 18:29:25 DEBUG "value": [ 18:29:25 DEBUG } 18:29:25 DEBUG }, 18:29:25 DEBUG "0" 18:29:25 DEBUG --- stdout --- 18:29:25 DEBUG ] 18:29:25 DEBUG "value": [ 18:29:25 DEBUG "0" 18:29:25 DEBUG ] 18:29:25 DEBUG ] 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG ] 18:29:25 DEBUG "value": [ 18:29:25 DEBUG ] 18:29:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 147m 3369Mi am-769b98f8d5-phmxx 123m 3622Mi am-769b98f8d5-zbtk2 116m 3358Mi ds-cts-0 12m 498Mi ds-cts-1 15m 481Mi ds-cts-2 8m 484Mi ds-idrepo-0 4016m 2472Mi ds-idrepo-1 3638m 2575Mi ds-idrepo-2 3604m 2463Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 8m 1093Mi idm-6c95d7588f-n2wtz 16m 1059Mi lodemon-596c8449d-bbnvt 1m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 196m 240Mi 18:29:25 DEBUG } 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG "0" 18:29:25 DEBUG } 18:29:25 DEBUG 1714066165, 18:29:25 DEBUG } 18:29:25 DEBUG --- stderr --- 18:29:25 DEBUG ] 18:29:25 DEBUG "0" 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG ] 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG "0" 18:29:25 DEBUG ] 18:29:25 DEBUG 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG ] 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 DEBUG } 18:29:25 INFO [loop_until]: OK (rc = 0) 18:29:25 DEBUG --- stdout --- 18:29:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3483m 21% 4670Mi 7% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3431m 21% 3763Mi 6% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4027m 25% 3687Mi 6% gke-xlou-cdm-default-pool-9d1e5395-qd5b 64m 0% 1811Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 173m 1% 5993Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 62m 0% 1685Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 68m 0% 1655Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 194m 1% 5009Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 79m 0% 2930Mi 4% gke-xlou-cdm-ds-174e6abc-k8cj 176m 1% 3651Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 168m 1% 5198Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 66m 0% 1402Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 65m 0% 2683Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 265m 1% 3788Mi 6% 18:29:25 DEBUG --- stderr --- 18:29:25 DEBUG 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 ERROR [http_cmd]: ERROR 18:30:25 ERROR --- status code --- 18:30:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:30:25 ERROR --- http response --- 18:30:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:30:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG "result": [ 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG { 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG "metric": { 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG }, 18:30:25 DEBUG { 18:30:25 DEBUG "value": [ 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG 18:30:25 DEBUG "data": { 18:30:25 DEBUG 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG 18:30:25 DEBUG "result": [ 18:30:25 DEBUG [...] 18:30:25 DEBUG { 18:30:25 DEBUG 18:30:25 DEBUG "metric": { 18:30:25 DEBUG 18:30:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:30:25 DEBUG 18:30:25 DEBUG }, 18:30:25 DEBUG "pod": "login-ui-5fdf49fc66-rzjdb" 18:30:25 DEBUG "value": [ 18:30:25 DEBUG }, 18:30:25 DEBUG 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG 18:30:25 DEBUG "0" 18:30:25 DEBUG [...] 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG 18:30:25 DEBUG } 18:30:25 DEBUG 18:30:25 DEBUG } 18:30:25 DEBUG 18:30:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0.21226255172204506" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "ds-idrepo-1" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "ds-idrepo-0" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "login-ui-5fdf49fc66-rzjdb" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0.00002928870292886934" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "instance": "10.142.0.24:9100" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 DEBUG 18:30:25 DEBUG "instance": "10.142.0.74:9100" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "8496.733333333334" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "ds-idrepo-1" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "ds-idrepo-0" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "instance": "10.142.0.24:9100" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "instance": "10.142.0.74:9100" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "4516600832" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- http response --- 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG { 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG { 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG "data": { 18:30:25 DEBUG "data": { 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "resultType": "vector", 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG "resultType": "vector", 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG "result": [ 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG { 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG { 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "metric": { 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG "metric": { 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG "pod": "ds-cts-0" 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:30:25 DEBUG "result": [ 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:25 DEBUG { 18:30:25 DEBUG }, 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG }, 18:30:25 DEBUG { 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "value": [ 18:30:25 DEBUG { 18:30:25 DEBUG "value": [ 18:30:25 DEBUG "metric": { 18:30:25 DEBUG { 18:30:25 DEBUG "instance": "10.142.0.24:9100" 18:30:25 DEBUG 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG 18:30:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG }, 18:30:25 DEBUG 18:30:25 DEBUG "data": { 18:30:25 DEBUG 18:30:25 DEBUG }, 18:30:25 DEBUG "data": { 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG 18:30:25 DEBUG "value": [ 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG "result": [ 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG "result": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG { 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG { 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "metric": { 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "metric": { 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG "instance": "10.142.0.24:9100" 18:30:25 DEBUG 18:30:25 DEBUG "pod": "ds-cts-2" 18:30:25 DEBUG }, 18:30:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:30:25 DEBUG 18:30:25 DEBUG }, 18:30:25 DEBUG 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG }, 18:30:25 DEBUG 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "instance": "10.142.0.74:9100" 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:30:25 DEBUG 18:30:25 DEBUG }, 18:30:25 DEBUG "0" 18:30:25 DEBUG 18:30:25 DEBUG "0" 18:30:25 DEBUG }, 18:30:25 DEBUG 18:30:25 DEBUG "value": [ 18:30:25 DEBUG ] 18:30:25 DEBUG [...] 18:30:25 DEBUG ] 18:30:25 DEBUG "value": [ 18:30:25 DEBUG [...] 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG } 18:30:25 DEBUG 18:30:25 DEBUG } 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG 18:30:25 DEBUG ] 18:30:25 DEBUG "0" 18:30:25 DEBUG 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG "instance": "10.142.0.74:9100" 18:30:25 DEBUG ] 18:30:25 DEBUG }, 18:30:25 DEBUG ] 18:30:25 DEBUG }, 18:30:25 DEBUG } 18:30:25 DEBUG "value": [ 18:30:25 DEBUG } 18:30:25 DEBUG "value": [ 18:30:25 DEBUG } 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG } 18:30:25 DEBUG 1714066225, 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 DEBUG "0" 18:30:25 DEBUG "0.000333333333333267" 18:30:25 DEBUG ] 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "ds-idrepo-1" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "ds-cts-2" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "ds-idrepo-1" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "ds-idrepo-0" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO 18:30:25 INFO [loop_until]: kubectl --namespace=xlou top pods 18:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "pod": "login-ui-5fdf49fc66-rzjdb" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "350425088" 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066225" 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG { 18:30:25 DEBUG { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "data": { 18:30:25 DEBUG "data": { 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "result": [ 18:30:25 DEBUG "result": [ 18:30:25 DEBUG { 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG { 18:30:25 DEBUG "metric": { 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "instance": "10.142.0.24:9100" 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:30:25 DEBUG }, 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG ----- output ----- 18:30:25 INFO [http_cmd]: http status code OK 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG { 18:30:25 DEBUG --- status code --- 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG "data": { 18:30:25 DEBUG --- http response --- 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG "result": [ 18:30:25 DEBUG ----- output ----- 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG { 18:30:25 DEBUG { 18:30:25 DEBUG { 18:30:25 DEBUG 18:30:25 DEBUG "instance": "10.142.0.74:9100" 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "metric": { 18:30:25 DEBUG "status": "success", 18:30:25 DEBUG "pod": "login-ui-5fdf49fc66-rzjdb" 18:30:25 DEBUG }, 18:30:25 DEBUG "data": { 18:30:25 DEBUG "pod": "ds-cts-0" 18:30:25 DEBUG "data": { 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG }, 18:30:25 DEBUG "resultType": "vector", 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "result": [ 18:30:25 DEBUG "value": [ 18:30:25 DEBUG "result": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "5415.166666666666" 18:30:25 DEBUG { 18:30:25 DEBUG 18:30:25 DEBUG { 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG "metric": { 18:30:25 INFO 18:30:25 DEBUG 18:30:25 DEBUG "metric": { 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG "pod": "ds-cts-0" 18:30:25 INFO [loop_until]: kubectl --namespace=xlou top node 18:30:25 DEBUG 18:30:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG }, 18:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:30:25 DEBUG [...] 18:30:25 DEBUG }, 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG "value": [ 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG } 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 INFO [loop_until]: OK (rc = 0) 18:30:25 DEBUG "pod": "ds-cts-2" 18:30:25 DEBUG 18:30:25 DEBUG [...] 18:30:25 DEBUG --- stdout --- 18:30:25 DEBUG }, 18:30:25 DEBUG [...] 18:30:25 DEBUG 18:30:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 145m 3369Mi am-769b98f8d5-phmxx 112m 3622Mi am-769b98f8d5-zbtk2 124m 3359Mi ds-cts-0 10m 498Mi ds-cts-1 13m 482Mi ds-cts-2 13m 495Mi ds-idrepo-0 4050m 2505Mi ds-idrepo-1 3686m 2586Mi ds-idrepo-2 3817m 2485Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 5m 1093Mi idm-6c95d7588f-n2wtz 6m 1060Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 187m 241Mi 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG --- stderr --- 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG 18:30:25 DEBUG "0" 18:30:25 DEBUG 18:30:25 DEBUG "pod": "ds-cts-2" 18:30:25 DEBUG ] 18:30:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:30:25 DEBUG }, 18:30:25 DEBUG } 18:30:25 DEBUG }, 18:30:25 DEBUG "value": [ 18:30:25 DEBUG ] 18:30:25 DEBUG "value": [ 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG } 18:30:25 DEBUG 1714066225, 18:30:25 DEBUG "0" 18:30:25 DEBUG } 18:30:25 DEBUG "0" 18:30:25 DEBUG ] 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG ] 18:30:25 DEBUG ] 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:25 DEBUG } 18:30:26 INFO [loop_until]: OK (rc = 0) 18:30:26 DEBUG --- stdout --- 18:30:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3766m 23% 4717Mi 8% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3624m 22% 3777Mi 6% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4115m 25% 3715Mi 6% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1807Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 182m 1% 5986Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 65m 0% 1695Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 65m 0% 1666Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 194m 1% 5011Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 71m 0% 2930Mi 4% gke-xlou-cdm-ds-174e6abc-k8cj 176m 1% 3658Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 162m 1% 5203Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 66m 0% 1387Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 66m 0% 2687Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 261m 1% 3785Mi 6% 18:30:26 DEBUG --- stderr --- 18:30:26 DEBUG 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066285" 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 ERROR [http_cmd]: ERROR 18:31:25 ERROR --- status code --- 18:31:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:31:25 ERROR --- http response --- 18:31:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:31:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG ----- output ----- 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG { 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG "data": { 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG "result": [ 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG { 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "pod": "ds-idrepo-1" 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG }, 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG "value": [ 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:31:25 DEBUG 18:31:25 DEBUG "pod": "ds-cts-0" 18:31:25 DEBUG }, 18:31:25 DEBUG [...] 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "ds-idrepo-0" 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG }, 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "0" 18:31:25 DEBUG 18:31:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:31:25 DEBUG ] 18:31:25 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:31:25 DEBUG }, 18:31:25 DEBUG } 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG ] 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG } 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0.21027726617811773" 18:31:25 DEBUG } 18:31:25 DEBUG "35142083.43566009" 18:31:25 DEBUG ] 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "ds-cts-0" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0.01011224457668199" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "instance": "10.142.0.24:9100" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "instance": "10.142.0.74:9100" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "8904.666666666666" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066285" 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG { 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG "data": { 18:31:25 DEBUG { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG "data": { 18:31:25 DEBUG { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "result": [ 18:31:25 DEBUG "pod": "ds-idrepo-1" 18:31:25 DEBUG { 18:31:25 DEBUG }, 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "value": [ 18:31:25 DEBUG "pod": "ds-cts-0" 18:31:25 DEBUG 18:31:25 DEBUG }, 18:31:25 DEBUG 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "ds-idrepo-0" 18:31:25 DEBUG 18:31:25 DEBUG }, 18:31:25 DEBUG 18:31:25 DEBUG "value": [ 18:31:25 DEBUG "pod": "ds-cts-2" 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG }, 18:31:25 DEBUG "0" 18:31:25 DEBUG "value": [ 18:31:25 DEBUG ] 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG } 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG ] 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG } 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:25 DEBUG } 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "instance": "10.142.0.24:9100" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "instance": "10.142.0.74:9100" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "4516873216" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "instance": "10.142.0.24:9100" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "instance": "10.142.0.74:9100" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066285" 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG { 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "data": { 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:31:25 DEBUG "result": [ 18:31:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:31:25 DEBUG }, 18:31:25 DEBUG { 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG "instance": "10.142.0.24:9100" 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG }, 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:31:25 DEBUG 18:31:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:31:25 DEBUG }, 18:31:25 DEBUG 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "instance": "10.142.0.74:9100" 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG }, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG "value": [ 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG "0.00020000000000000757" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "ds-idrepo-1" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "ds-cts-2" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "ds-idrepo-1" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "ds-idrepo-0" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066285" 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 INFO [http_cmd]: http status code OK 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG --- status code --- 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG --- http response --- 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:31:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG ----- output ----- 18:31:25 DEBUG { 18:31:25 DEBUG { 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "status": "success", 18:31:25 DEBUG "data": { 18:31:25 DEBUG "data": { 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "resultType": "vector", 18:31:25 DEBUG "result": [ 18:31:25 DEBUG "result": [ 18:31:25 DEBUG { 18:31:25 DEBUG { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "metric": { 18:31:25 DEBUG "pod": "ds-cts-0" 18:31:25 DEBUG "pod": "ds-cts-0" 18:31:25 DEBUG }, 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG [...] 18:31:25 DEBUG [...] 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG 18:31:25 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:31:25 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:31:25 DEBUG }, 18:31:25 DEBUG }, 18:31:25 DEBUG "value": [ 18:31:25 DEBUG "value": [ 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG 1714066285, 18:31:25 DEBUG "515.8527754163912" 18:31:25 DEBUG "0" 18:31:25 DEBUG ] 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG ] 18:31:25 DEBUG ] 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:25 DEBUG } 18:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066286" 18:31:26 INFO [http_cmd]: http status code OK 18:31:26 DEBUG --- status code --- 18:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:26 DEBUG --- http response --- 18:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:26 DEBUG ----- output ----- 18:31:26 DEBUG { 18:31:26 DEBUG "status": "success", 18:31:26 DEBUG "data": { 18:31:26 DEBUG "resultType": "vector", 18:31:26 DEBUG "result": [ 18:31:26 DEBUG { 18:31:26 DEBUG "metric": { 18:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066286" 18:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066286" 18:31:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:31:26 DEBUG }, 18:31:26 DEBUG "value": [ 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG [...] 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:31:26 DEBUG }, 18:31:26 DEBUG "value": [ 18:31:26 DEBUG 1714066286, 18:31:26 DEBUG "351166464" 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG } 18:31:26 INFO [http_cmd]: http status code OK 18:31:26 INFO [http_cmd]: http status code OK 18:31:26 DEBUG --- status code --- 18:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:26 DEBUG --- status code --- 18:31:26 DEBUG --- http response --- 18:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:31:26 DEBUG --- http response --- 18:31:26 DEBUG ----- output ----- 18:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:26 DEBUG { 18:31:26 DEBUG ----- output ----- 18:31:26 DEBUG "status": "success", 18:31:26 DEBUG { 18:31:26 DEBUG "data": { 18:31:26 DEBUG "status": "success", 18:31:26 DEBUG "resultType": "vector", 18:31:26 DEBUG "data": { 18:31:26 DEBUG "result": [ 18:31:26 DEBUG "resultType": "vector", 18:31:26 DEBUG { 18:31:26 DEBUG "result": [ 18:31:26 DEBUG "metric": { 18:31:26 DEBUG { 18:31:26 DEBUG "pod": "ds-cts-0" 18:31:26 DEBUG "metric": { 18:31:26 DEBUG }, 18:31:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:31:26 DEBUG "value": [ 18:31:26 DEBUG }, 18:31:26 DEBUG 18:31:26 DEBUG "value": [ 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG [...] 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG [...] 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG "pod": "ds-cts-2" 18:31:26 DEBUG 18:31:26 DEBUG }, 18:31:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:31:26 DEBUG "value": [ 18:31:26 DEBUG }, 18:31:26 DEBUG 1714066286, 18:31:26 DEBUG "value": [ 18:31:26 DEBUG "0" 18:31:26 DEBUG 1714066286, 18:31:26 DEBUG ] 18:31:26 DEBUG "0" 18:31:26 DEBUG } 18:31:26 DEBUG ] 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG } 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG } 18:31:26 DEBUG } 18:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066286" 18:31:26 INFO 18:31:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:31:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:31:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066286" 18:31:26 INFO [http_cmd]: http status code OK 18:31:26 DEBUG --- status code --- 18:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:26 DEBUG --- http response --- 18:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:26 DEBUG ----- output ----- 18:31:26 DEBUG { 18:31:26 DEBUG "status": "success", 18:31:26 DEBUG "data": { 18:31:26 DEBUG "resultType": "vector", 18:31:26 DEBUG "result": [ 18:31:26 DEBUG { 18:31:26 DEBUG "metric": { 18:31:26 DEBUG "instance": "10.142.0.24:9100" 18:31:26 DEBUG }, 18:31:26 DEBUG "value": [ 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG [...] 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG "instance": "10.142.0.74:9100" 18:31:26 DEBUG }, 18:31:26 DEBUG "value": [ 18:31:26 DEBUG 1714066286, 18:31:26 DEBUG "4979.8" 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG } 18:31:26 INFO [http_cmd]: http status code OK 18:31:26 DEBUG --- status code --- 18:31:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:26 DEBUG --- http response --- 18:31:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:31:26 DEBUG ----- output ----- 18:31:26 DEBUG { 18:31:26 DEBUG "status": "success", 18:31:26 DEBUG "data": { 18:31:26 DEBUG "resultType": "vector", 18:31:26 DEBUG "result": [ 18:31:26 DEBUG { 18:31:26 DEBUG "metric": { 18:31:26 DEBUG "pod": "ds-cts-0" 18:31:26 DEBUG }, 18:31:26 DEBUG "value": [ 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG [...] 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG 18:31:26 DEBUG "pod": "ds-cts-2" 18:31:26 DEBUG }, 18:31:26 DEBUG "value": [ 18:31:26 DEBUG 1714066286, 18:31:26 DEBUG "0" 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG ] 18:31:26 DEBUG } 18:31:26 DEBUG } 18:31:26 INFO [loop_until]: OK (rc = 0) 18:31:26 DEBUG --- stdout --- 18:31:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 142m 3369Mi am-769b98f8d5-phmxx 122m 3623Mi am-769b98f8d5-zbtk2 120m 3359Mi ds-cts-0 9m 499Mi ds-cts-1 12m 482Mi ds-cts-2 11m 495Mi ds-idrepo-0 4206m 2894Mi ds-idrepo-1 3670m 2649Mi ds-idrepo-2 3908m 2548Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 8m 1094Mi idm-6c95d7588f-n2wtz 9m 1060Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 179m 241Mi 18:31:26 DEBUG --- stderr --- 18:31:26 DEBUG 18:31:26 INFO 18:31:26 INFO [loop_until]: kubectl --namespace=xlou top node 18:31:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:31:26 INFO [loop_until]: OK (rc = 0) 18:31:26 DEBUG --- stdout --- 18:31:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3949m 24% 4791Mi 8% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3782m 23% 3859Mi 6% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4193m 26% 4102Mi 6% gke-xlou-cdm-default-pool-9d1e5395-qd5b 59m 0% 1814Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 173m 1% 5989Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 62m 0% 1703Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 65m 0% 1651Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 192m 1% 5004Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 72m 0% 2936Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 176m 1% 3656Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 171m 1% 5205Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1387Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 63m 0% 2697Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 249m 1% 3783Mi 6% 18:31:26 DEBUG --- stderr --- 18:31:26 DEBUG 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066345" 18:32:25 ERROR [http_cmd]: ERROR 18:32:25 ERROR --- status code --- 18:32:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:32:25 ERROR --- http response --- 18:32:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:32:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "0.17672143824242073" 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066345" 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "0.20303686928039683" 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "pod": "ds-idrepo-1" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:25 DEBUG 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG 18:32:25 DEBUG { 18:32:25 DEBUG "pod": "ds-idrepo-0" 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG }, 18:32:25 DEBUG "data": { 18:32:25 DEBUG "value": [ 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "result": [ 18:32:25 DEBUG "0" 18:32:25 DEBUG { 18:32:25 DEBUG ] 18:32:25 DEBUG "metric": { 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG "instance": "10.142.0.24:9100" 18:32:25 DEBUG ] 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG }, 18:32:25 DEBUG } 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG "value": [ 18:32:25 DEBUG } 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:32:25 DEBUG 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG 18:32:25 DEBUG { 18:32:25 DEBUG [...] 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG 18:32:25 DEBUG "data": { 18:32:25 DEBUG 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG 18:32:25 DEBUG "result": [ 18:32:25 DEBUG "instance": "10.142.0.74:9100" 18:32:25 DEBUG { 18:32:25 DEBUG }, 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "value": [ 18:32:25 DEBUG "pod": "ds-cts-0" 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG }, 18:32:25 DEBUG "8436.666666666666" 18:32:25 DEBUG "value": [ 18:32:25 DEBUG ] 18:32:25 DEBUG 18:32:25 DEBUG } 18:32:25 DEBUG 18:32:25 DEBUG ] 18:32:25 DEBUG 18:32:25 DEBUG } 18:32:25 DEBUG [...] 18:32:25 DEBUG } 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "pod": "ds-cts-2" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "0" 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "14766.030376455292" 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "0" 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "0" 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066345" 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "pod": "ds-idrepo-1" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "pod": "ds-idrepo-0" 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "0" 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 DEBUG { 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG "data": { 18:32:25 DEBUG "data": { 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG "result": [ 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "metric": { 18:32:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:32:25 DEBUG "instance": "10.142.0.24:9100" 18:32:25 DEBUG }, 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG [...] 18:32:25 DEBUG [...] 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG 18:32:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:32:25 DEBUG "instance": "10.142.0.74:9100" 18:32:25 DEBUG }, 18:32:25 DEBUG }, 18:32:25 DEBUG "value": [ 18:32:25 DEBUG "value": [ 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG 1714066345, 18:32:25 DEBUG "0" 18:32:25 DEBUG "4517114880" 18:32:25 DEBUG ] 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 DEBUG ] 18:32:25 DEBUG ] 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 DEBUG } 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066345" 18:32:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066345" 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG { 18:32:25 INFO [http_cmd]: http status code OK 18:32:25 DEBUG "status": "success", 18:32:25 DEBUG --- status code --- 18:32:25 DEBUG "data": { 18:32:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:25 DEBUG "resultType": "vector", 18:32:25 DEBUG --- http response --- 18:32:25 DEBUG "result": [ 18:32:25 DEBUG { 18:32:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:25 DEBUG "metric": { 18:32:25 DEBUG ----- output ----- 18:32:25 DEBUG "instance": "10.142.0.24:9100" 18:32:26 DEBUG { 18:32:26 DEBUG }, 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "value": [ 18:32:26 DEBUG "data": { 18:32:26 DEBUG 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG 18:32:26 DEBUG "result": [ 18:32:26 DEBUG 18:32:26 DEBUG { 18:32:26 DEBUG [...] 18:32:26 DEBUG "metric": { 18:32:26 DEBUG 18:32:26 DEBUG "instance": "10.142.0.24:9100" 18:32:26 DEBUG 18:32:26 DEBUG }, 18:32:26 DEBUG 18:32:26 DEBUG "value": [ 18:32:26 DEBUG "instance": "10.142.0.74:9100" 18:32:26 DEBUG 18:32:26 DEBUG }, 18:32:26 DEBUG 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 1714066345, 18:32:26 DEBUG [...] 18:32:26 DEBUG "0.00006666666666674815" 18:32:26 DEBUG 18:32:26 DEBUG ] 18:32:26 DEBUG 18:32:26 DEBUG } 18:32:26 DEBUG 18:32:26 DEBUG ] 18:32:26 DEBUG "instance": "10.142.0.74:9100" 18:32:26 DEBUG } 18:32:26 DEBUG }, 18:32:26 DEBUG } 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066345, 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:32:26 DEBUG "pod": "ds-idrepo-1" 18:32:26 DEBUG }, 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:32:26 DEBUG "pod": "ds-cts-2" 18:32:26 DEBUG }, 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "ds-idrepo-1" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "ds-idrepo-0" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:32:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:32:26 DEBUG }, 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:32:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:32:26 DEBUG }, 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066346" 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066346" 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "instance": "10.142.0.24:9100" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "instance": "10.142.0.74:9100" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "4120.933333333334" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "352206848" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO 18:32:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:32:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "ds-cts-0" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "ds-cts-2" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066346" 18:32:26 INFO [http_cmd]: http status code OK 18:32:26 DEBUG --- status code --- 18:32:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:26 DEBUG --- http response --- 18:32:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:32:26 DEBUG ----- output ----- 18:32:26 DEBUG { 18:32:26 DEBUG "status": "success", 18:32:26 DEBUG "data": { 18:32:26 DEBUG "resultType": "vector", 18:32:26 DEBUG "result": [ 18:32:26 DEBUG { 18:32:26 DEBUG "metric": { 18:32:26 DEBUG "pod": "ds-cts-0" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG [...] 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG 18:32:26 DEBUG "pod": "ds-cts-2" 18:32:26 DEBUG }, 18:32:26 DEBUG "value": [ 18:32:26 DEBUG 1714066346, 18:32:26 DEBUG "0" 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG ] 18:32:26 DEBUG } 18:32:26 DEBUG } 18:32:26 INFO [loop_until]: OK (rc = 0) 18:32:26 DEBUG --- stdout --- 18:32:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 142m 3369Mi am-769b98f8d5-phmxx 111m 3624Mi am-769b98f8d5-zbtk2 116m 3361Mi ds-cts-0 8m 499Mi ds-cts-1 9m 482Mi ds-cts-2 9m 495Mi ds-idrepo-0 3890m 2882Mi ds-idrepo-1 3548m 2970Mi ds-idrepo-2 3732m 2829Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 8m 1095Mi idm-6c95d7588f-n2wtz 6m 1061Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 179m 242Mi 18:32:26 DEBUG --- stderr --- 18:32:26 DEBUG 18:32:26 INFO 18:32:26 INFO [loop_until]: kubectl --namespace=xlou top node 18:32:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:32:26 INFO [loop_until]: OK (rc = 0) 18:32:26 DEBUG --- stdout --- 18:32:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3710m 23% 5046Mi 8% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3479m 21% 4176Mi 7% gke-xlou-cdm-default-pool-9d1e5395-jnd9 3911m 24% 4097Mi 6% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1811Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 170m 1% 5989Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 59m 0% 1702Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 61m 0% 1653Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 190m 1% 5005Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 69m 0% 2937Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 162m 1% 3656Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 165m 1% 5213Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 64m 0% 1384Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 68m 0% 2691Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 251m 1% 3787Mi 6% 18:32:26 DEBUG --- stderr --- 18:32:26 DEBUG 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066405" 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066405" 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066405" 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066405" 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066405" 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066405" 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 ERROR [http_cmd]: ERROR 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066405" 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 DEBUG --- status code --- 18:33:25 DEBUG --- status code --- 18:33:25 ERROR --- status code --- 18:33:25 DEBUG --- status code --- 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 DEBUG --- http response --- 18:33:25 DEBUG --- http response --- 18:33:25 ERROR --- http response --- 18:33:25 DEBUG --- http response --- 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066405" 18:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:25 DEBUG ----- output ----- 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 DEBUG ----- output ----- 18:33:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:33:25 DEBUG ----- output ----- 18:33:25 DEBUG { 18:33:25 DEBUG --- status code --- 18:33:25 DEBUG { 18:33:25 DEBUG { 18:33:25 DEBUG "status": "success", 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 DEBUG "status": "success", 18:33:25 DEBUG "status": "success", 18:33:25 DEBUG "data": { 18:33:25 DEBUG --- http response --- 18:33:25 DEBUG "data": { 18:33:25 DEBUG "data": { 18:33:25 DEBUG "resultType": "vector", 18:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:25 DEBUG "resultType": "vector", 18:33:25 DEBUG "resultType": "vector", 18:33:25 DEBUG "result": [ 18:33:25 DEBUG ----- output ----- 18:33:25 DEBUG "result": [ 18:33:25 DEBUG "result": [ 18:33:25 DEBUG { 18:33:25 DEBUG { 18:33:25 DEBUG { 18:33:25 DEBUG { 18:33:25 DEBUG "metric": { 18:33:25 DEBUG "status": "success", 18:33:25 DEBUG "metric": { 18:33:25 DEBUG "metric": { 18:33:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:33:25 DEBUG "data": { 18:33:25 DEBUG "instance": "10.142.0.24:9100" 18:33:25 DEBUG "pod": "ds-idrepo-1" 18:33:25 DEBUG }, 18:33:25 DEBUG "resultType": "vector", 18:33:25 DEBUG }, 18:33:25 DEBUG }, 18:33:25 DEBUG "value": [ 18:33:25 DEBUG "result": [ 18:33:25 DEBUG "value": [ 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 18:33:25 DEBUG { 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG "metric": { 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG [...] 18:33:25 DEBUG }, 18:33:25 DEBUG [...] 18:33:25 DEBUG [...] 18:33:25 DEBUG 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:33:25 DEBUG 18:33:25 DEBUG "instance": "10.142.0.74:9100" 18:33:25 DEBUG "pod": "ds-idrepo-0" 18:33:25 DEBUG }, 18:33:25 DEBUG [...] 18:33:25 DEBUG }, 18:33:25 DEBUG }, 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 18:33:25 DEBUG "value": [ 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 1714066405, 18:33:25 DEBUG 18:33:25 DEBUG 1714066405, 18:33:25 DEBUG 1714066405, 18:33:25 DEBUG "0.19130701066033987" 18:33:25 DEBUG 18:33:25 DEBUG "10209.766666666666" 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 DEBUG "0" 18:33:25 DEBUG ] 18:33:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:33:25 DEBUG ] 18:33:25 DEBUG --- status code --- 18:33:25 DEBUG --- status code --- 18:33:25 DEBUG ] 18:33:25 DEBUG } 18:33:25 DEBUG }, 18:33:25 DEBUG } 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 DEBUG } 18:33:25 DEBUG ] 18:33:25 DEBUG "value": [ 18:33:25 DEBUG ] 18:33:25 DEBUG --- http response --- 18:33:25 DEBUG --- http response --- 18:33:25 DEBUG ] 18:33:25 DEBUG } 18:33:25 DEBUG 1714066405, 18:33:25 DEBUG } 18:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:33:25 DEBUG } 18:33:25 DEBUG } 18:33:25 DEBUG "0.21858458805566344" 18:33:25 DEBUG } 18:33:25 DEBUG ----- output ----- 18:33:25 DEBUG ----- output ----- 18:33:25 DEBUG } 18:33:25 DEBUG ] 18:33:25 DEBUG { 18:33:25 DEBUG { 18:33:25 DEBUG } 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 DEBUG "status": "success", 18:33:25 DEBUG "status": "success", 18:33:25 DEBUG ] 18:33:25 DEBUG --- status code --- 18:33:25 DEBUG "data": { 18:33:25 DEBUG "data": { 18:33:25 DEBUG } 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 DEBUG "resultType": "vector", 18:33:25 DEBUG "resultType": "vector", 18:33:25 DEBUG } 18:33:25 DEBUG --- http response --- 18:33:25 DEBUG "result": [ 18:33:25 DEBUG "result": [ 18:33:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:33:25 DEBUG { 18:33:25 DEBUG { 18:33:25 DEBUG "metric": { 18:33:25 DEBUG "metric": { 18:33:25 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:33:25 DEBUG "pod": "ds-cts-0" 18:33:25 DEBUG }, 18:33:25 DEBUG }, 18:33:25 DEBUG "value": [ 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG [...] 18:33:25 DEBUG [...] 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG "pod": "am-769b98f8d5-phmxx" 18:33:25 DEBUG "pod": "ds-cts-2" 18:33:25 DEBUG }, 18:33:25 DEBUG }, 18:33:25 DEBUG "value": [ 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 1714066405, 18:33:25 DEBUG 1714066405, 18:33:25 DEBUG "0" 18:33:25 DEBUG "0" 18:33:25 DEBUG ] 18:33:25 DEBUG ] 18:33:25 DEBUG } 18:33:25 DEBUG } 18:33:25 DEBUG ] 18:33:25 DEBUG ] 18:33:25 DEBUG } 18:33:25 DEBUG } 18:33:25 DEBUG } 18:33:25 DEBUG } 18:33:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066405" 18:33:25 INFO [http_cmd]: http status code OK 18:33:25 DEBUG --- status code --- 18:33:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:25 DEBUG --- http response --- 18:33:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:33:25 DEBUG ----- output ----- 18:33:25 DEBUG { 18:33:25 DEBUG "status": "success", 18:33:25 DEBUG "data": { 18:33:25 DEBUG "resultType": "vector", 18:33:25 DEBUG "result": [ 18:33:25 DEBUG { 18:33:25 DEBUG "metric": { 18:33:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:33:25 DEBUG }, 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG [...] 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG 18:33:25 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:33:25 DEBUG }, 18:33:25 DEBUG "value": [ 18:33:25 DEBUG 1714066405, 18:33:25 DEBUG "5612.083945996525" 18:33:25 DEBUG ] 18:33:25 DEBUG } 18:33:25 DEBUG ] 18:33:25 DEBUG } 18:33:25 DEBUG } 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "0" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066406" 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "pod": "ds-idrepo-1" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "ds-idrepo-0" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "0" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "instance": "10.142.0.24:9100" 18:33:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:33:26 DEBUG }, 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "instance": "10.142.0.74:9100" 18:33:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:33:26 DEBUG }, 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "4517362688" 18:33:26 DEBUG "0" 18:33:26 DEBUG ] 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG } 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:33:26 DEBUG }, 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG "value": [ 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:26 DEBUG 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG 18:33:26 DEBUG { 18:33:26 DEBUG 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:33:26 DEBUG "data": { 18:33:26 DEBUG }, 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "value": [ 18:33:26 DEBUG "result": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG { 18:33:26 DEBUG "0" 18:33:26 DEBUG "metric": { 18:33:26 DEBUG ] 18:33:26 DEBUG "instance": "10.142.0.24:9100" 18:33:26 DEBUG } 18:33:26 DEBUG }, 18:33:26 DEBUG ] 18:33:26 DEBUG "value": [ 18:33:26 DEBUG } 18:33:26 DEBUG 18:33:26 DEBUG } 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "instance": "10.142.0.74:9100" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "0.00013333333333325942" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066406" 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:33:26 DEBUG { 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG { 18:33:26 DEBUG "data": { 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG "data": { 18:33:26 DEBUG "result": [ 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG { 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG "result": [ 18:33:26 DEBUG "metric": { 18:33:26 DEBUG { 18:33:26 DEBUG { 18:33:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "metric": { 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 DEBUG }, 18:33:26 DEBUG "data": { 18:33:26 DEBUG "pod": "ds-idrepo-1" 18:33:26 DEBUG "value": [ 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG }, 18:33:26 DEBUG 18:33:26 DEBUG "result": [ 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG { 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "metric": { 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG "instance": "10.142.0.24:9100" 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG }, 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG }, 18:33:26 DEBUG 18:33:26 DEBUG "pod": "ds-cts-2" 18:33:26 DEBUG "value": [ 18:33:26 DEBUG [...] 18:33:26 DEBUG }, 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG 18:33:26 DEBUG "value": [ 18:33:26 DEBUG "0" 18:33:26 DEBUG 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG ] 18:33:26 DEBUG 18:33:26 DEBUG "0" 18:33:26 DEBUG } 18:33:26 DEBUG "instance": "10.142.0.74:9100" 18:33:26 DEBUG ] 18:33:26 DEBUG ] 18:33:26 DEBUG }, 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 DEBUG "value": [ 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG } 18:33:26 DEBUG "0" 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "0" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "instance": "10.142.0.24:9100" 18:33:26 DEBUG }, 18:33:26 DEBUG "pod": "ds-idrepo-1" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:33:26 DEBUG 18:33:26 DEBUG "instance": "10.142.0.74:9100" 18:33:26 DEBUG }, 18:33:26 DEBUG "pod": "ds-idrepo-0" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "0" 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "4548.566666666667" 18:33:26 DEBUG ] 18:33:26 DEBUG "0" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "0" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG { 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG "data": { 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "data": { 18:33:26 DEBUG "pod": "ds-cts-0" 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG }, 18:33:26 DEBUG "result": [ 18:33:26 DEBUG "value": [ 18:33:26 DEBUG { 18:33:26 DEBUG 18:33:26 DEBUG "metric": { 18:33:26 DEBUG 18:33:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:33:26 DEBUG 18:33:26 DEBUG }, 18:33:26 DEBUG [...] 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "ds-cts-2" 18:33:26 DEBUG [...] 18:33:26 DEBUG }, 18:33:26 DEBUG 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG 18:33:26 DEBUG "0" 18:33:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:33:26 DEBUG ] 18:33:26 DEBUG }, 18:33:26 DEBUG } 18:33:26 DEBUG "value": [ 18:33:26 DEBUG ] 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG } 18:33:26 DEBUG "0" 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066406" 18:33:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066406" 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "353046528" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO [http_cmd]: http status code OK 18:33:26 DEBUG --- status code --- 18:33:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:26 DEBUG --- http response --- 18:33:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:33:26 DEBUG ----- output ----- 18:33:26 DEBUG { 18:33:26 DEBUG "status": "success", 18:33:26 DEBUG "data": { 18:33:26 DEBUG "resultType": "vector", 18:33:26 DEBUG "result": [ 18:33:26 DEBUG { 18:33:26 DEBUG "metric": { 18:33:26 DEBUG "pod": "ds-cts-0" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG [...] 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG 18:33:26 DEBUG "pod": "ds-cts-2" 18:33:26 DEBUG }, 18:33:26 DEBUG "value": [ 18:33:26 DEBUG 1714066406, 18:33:26 DEBUG "0" 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG ] 18:33:26 DEBUG } 18:33:26 DEBUG } 18:33:26 INFO 18:33:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:33:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:33:26 INFO [loop_until]: OK (rc = 0) 18:33:26 DEBUG --- stdout --- 18:33:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 149m 3371Mi am-769b98f8d5-phmxx 124m 3618Mi am-769b98f8d5-zbtk2 122m 3363Mi ds-cts-0 12m 500Mi ds-cts-1 10m 484Mi ds-cts-2 7m 496Mi ds-idrepo-0 4401m 3090Mi ds-idrepo-1 3946m 3059Mi ds-idrepo-2 3942m 3021Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 7m 1098Mi idm-6c95d7588f-n2wtz 6m 1061Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 193m 242Mi 18:33:26 DEBUG --- stderr --- 18:33:26 DEBUG 18:33:26 INFO 18:33:26 INFO [loop_until]: kubectl --namespace=xlou top node 18:33:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:33:26 INFO [loop_until]: OK (rc = 0) 18:33:26 DEBUG --- stdout --- 18:33:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 4053m 25% 5235Mi 8% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3832m 24% 4261Mi 7% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4409m 27% 4305Mi 7% gke-xlou-cdm-default-pool-9d1e5395-qd5b 63m 0% 1805Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 179m 1% 5991Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 64m 0% 1693Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 64m 0% 1650Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 205m 1% 5004Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 69m 0% 2942Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 181m 1% 3647Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 160m 1% 5207Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 62m 0% 1389Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 62m 0% 2686Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 262m 1% 3785Mi 6% 18:33:26 DEBUG --- stderr --- 18:33:26 DEBUG 18:34:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066465" 18:34:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066465" 18:34:25 ERROR [http_cmd]: ERROR 18:34:25 INFO [http_cmd]: http status code OK 18:34:25 ERROR --- status code --- 18:34:25 DEBUG --- status code --- 18:34:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:34:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:25 ERROR --- http response --- 18:34:25 DEBUG --- http response --- 18:34:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:34:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:34:25 DEBUG ----- output ----- 18:34:25 DEBUG { 18:34:25 DEBUG "status": "success", 18:34:25 DEBUG "data": { 18:34:25 DEBUG "resultType": "vector", 18:34:25 DEBUG "result": [ 18:34:25 DEBUG { 18:34:25 DEBUG "metric": { 18:34:25 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:34:25 DEBUG }, 18:34:25 DEBUG "value": [ 18:34:25 DEBUG 18:34:25 DEBUG 18:34:25 DEBUG 18:34:25 DEBUG [...] 18:34:25 DEBUG 18:34:25 DEBUG 18:34:25 DEBUG 18:34:25 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:34:25 DEBUG }, 18:34:25 DEBUG "value": [ 18:34:25 DEBUG 1714066465, 18:34:25 DEBUG "0.006293306112348541" 18:34:25 DEBUG ] 18:34:25 DEBUG } 18:34:25 DEBUG ] 18:34:25 DEBUG } 18:34:25 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "ds-cts-0" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "ds-cts-2" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG ] 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG } 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG ] 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG } 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG } 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0.21609159066154512" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "data": { 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:34:26 DEBUG "pod": "ds-idrepo-1" 18:34:26 DEBUG "instance": "10.142.0.24:9100" 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG [...] 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:34:26 DEBUG "pod": "ds-idrepo-0" 18:34:26 DEBUG "instance": "10.142.0.74:9100" 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 DEBUG "value": [ 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG "0" 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG "9960.8" 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:34:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:34:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG "34564624.59787181" 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "ds-idrepo-1" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "ds-idrepo-0" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "instance": "10.142.0.24:9100" 18:34:26 DEBUG "data": { 18:34:26 DEBUG "instance": "10.142.0.24:9100" 18:34:26 DEBUG }, 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG { 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "metric": { 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "ds-idrepo-1" 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG }, 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "instance": "10.142.0.74:9100" 18:34:26 DEBUG 18:34:26 DEBUG "instance": "10.142.0.74:9100" 18:34:26 DEBUG }, 18:34:26 DEBUG [...] 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0.00020000000000000757" 18:34:26 DEBUG 18:34:26 DEBUG "0" 18:34:26 DEBUG ] 18:34:26 DEBUG "pod": "ds-cts-2" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG }, 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG "value": [ 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG "0" 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG "data": { 18:34:26 DEBUG "data": { 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "data": { 18:34:26 DEBUG "instance": "10.142.0.24:9100" 18:34:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG { 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "metric": { 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG }, 18:34:26 DEBUG [...] 18:34:26 DEBUG [...] 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "instance": "10.142.0.74:9100" 18:34:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:34:26 DEBUG [...] 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG 18:34:26 DEBUG "4517577728" 18:34:26 DEBUG "0" 18:34:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG }, 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG "value": [ 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG "0" 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "data": { 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:34:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:34:26 DEBUG "metric": { 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG }, 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:34:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:34:26 DEBUG 18:34:26 DEBUG }, 18:34:26 DEBUG }, 18:34:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "value": [ 18:34:26 DEBUG }, 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "0" 18:34:26 DEBUG "0" 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG "506.80524622618157" 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "ds-idrepo-1" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "ds-idrepo-0" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG { 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:26 DEBUG "data": { 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG { 18:34:26 DEBUG "result": [ 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG { 18:34:26 DEBUG "data": { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:34:26 DEBUG "result": [ 18:34:26 DEBUG }, 18:34:26 DEBUG { 18:34:26 DEBUG "value": [ 18:34:26 DEBUG "metric": { 18:34:26 DEBUG 18:34:26 DEBUG "instance": "10.142.0.24:9100" 18:34:26 DEBUG 18:34:26 DEBUG }, 18:34:26 DEBUG 18:34:26 DEBUG "value": [ 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:34:26 DEBUG 18:34:26 DEBUG }, 18:34:26 DEBUG 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "instance": "10.142.0.74:9100" 18:34:26 DEBUG "0" 18:34:26 DEBUG }, 18:34:26 DEBUG ] 18:34:26 DEBUG "value": [ 18:34:26 DEBUG } 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG ] 18:34:26 DEBUG "4147.133333333333" 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "ds-cts-0" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "ds-cts-2" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "354693120" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066466" 18:34:26 INFO [http_cmd]: http status code OK 18:34:26 DEBUG --- status code --- 18:34:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:26 DEBUG --- http response --- 18:34:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:34:26 DEBUG ----- output ----- 18:34:26 DEBUG { 18:34:26 DEBUG "status": "success", 18:34:26 DEBUG "data": { 18:34:26 DEBUG "resultType": "vector", 18:34:26 DEBUG "result": [ 18:34:26 DEBUG { 18:34:26 DEBUG "metric": { 18:34:26 DEBUG "pod": "ds-cts-0" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG [...] 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG 18:34:26 DEBUG "pod": "ds-cts-2" 18:34:26 DEBUG }, 18:34:26 DEBUG "value": [ 18:34:26 DEBUG 1714066466, 18:34:26 DEBUG "0" 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG ] 18:34:26 DEBUG } 18:34:26 DEBUG } 18:34:26 INFO 18:34:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:34:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:34:26 INFO [loop_until]: OK (rc = 0) 18:34:26 DEBUG --- stdout --- 18:34:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 153m 3372Mi am-769b98f8d5-phmxx 122m 3625Mi am-769b98f8d5-zbtk2 123m 3363Mi ds-cts-0 8m 501Mi ds-cts-1 10m 484Mi ds-cts-2 10m 496Mi ds-idrepo-0 4316m 3276Mi ds-idrepo-1 3887m 3520Mi ds-idrepo-2 4001m 3259Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 8m 1098Mi idm-6c95d7588f-n2wtz 6m 1061Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 192m 242Mi 18:34:26 DEBUG --- stderr --- 18:34:26 DEBUG 18:34:26 INFO 18:34:26 INFO [loop_until]: kubectl --namespace=xlou top node 18:34:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:34:26 INFO [loop_until]: OK (rc = 0) 18:34:26 DEBUG --- stdout --- 18:34:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3973m 25% 5486Mi 9% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3649m 22% 4696Mi 8% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4325m 27% 4498Mi 7% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1802Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 178m 1% 5993Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 67m 0% 1699Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 62m 0% 1648Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 202m 1% 5017Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 67m 0% 2937Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 159m 1% 3647Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 182m 1% 5198Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 65m 0% 1401Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 64m 0% 2684Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 264m 1% 3785Mi 6% 18:34:26 DEBUG --- stderr --- 18:34:26 DEBUG 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066526" 18:35:26 ERROR [http_cmd]: ERROR 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 ERROR --- status code --- 18:35:26 DEBUG --- status code --- 18:35:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 ERROR --- http response --- 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:35:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG "data": { 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG "result": [ 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:35:26 DEBUG { 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG "metric": { 18:35:26 DEBUG { 18:35:26 DEBUG { 18:35:26 DEBUG "pod": "ds-cts-0" 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG }, 18:35:26 DEBUG "data": { 18:35:26 DEBUG "data": { 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG 18:35:26 DEBUG "result": [ 18:35:26 DEBUG "result": [ 18:35:26 DEBUG 18:35:26 DEBUG { 18:35:26 DEBUG { 18:35:26 DEBUG 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG [...] 18:35:26 DEBUG "instance": "10.142.0.24:9100" 18:35:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:35:26 DEBUG 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "ds-cts-2" 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG }, 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "value": [ 18:35:26 DEBUG [...] 18:35:26 DEBUG [...] 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "0" 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG ] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG } 18:35:26 DEBUG "instance": "10.142.0.74:9100" 18:35:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:35:26 DEBUG ] 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG } 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG } 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "9286.099999999999" 18:35:26 DEBUG "0.009400522521330999" 18:35:26 DEBUG ] 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0.20096150514905106" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG { 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG { 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG "data": { 18:35:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:35:26 DEBUG "data": { 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG "result": [ 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG { 18:35:26 DEBUG { 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "data": { 18:35:26 DEBUG "data": { 18:35:26 DEBUG "pod": "ds-idrepo-1" 18:35:26 DEBUG "pod": "ds-idrepo-1" 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG "result": [ 18:35:26 DEBUG "result": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG { 18:35:26 DEBUG { 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "ds-idrepo-1" 18:35:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG [...] 18:35:26 DEBUG [...] 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "ds-idrepo-0" 18:35:26 DEBUG "pod": "ds-cts-2" 18:35:26 DEBUG [...] 18:35:26 DEBUG [...] 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "0" 18:35:26 DEBUG "0" 18:35:26 DEBUG "pod": "ds-idrepo-0" 18:35:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:35:26 DEBUG ] 18:35:26 DEBUG ] 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG ] 18:35:26 DEBUG ] 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG "0" 18:35:26 DEBUG "31155.499702361267" 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "instance": "10.142.0.24:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "instance": "10.142.0.74:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "4517856256" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "instance": "10.142.0.24:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "instance": "10.142.0.74:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0.00030000000000001136" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG } 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "instance": "10.142.0.24:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "instance": "10.142.0.74:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "541.8347774323698" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:35:26 DEBUG }, 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG { 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "data": { 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG "result": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG { 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "metric": { 18:35:26 DEBUG 18:35:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:35:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:35:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG [...] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 DEBUG 18:35:26 DEBUG } 18:35:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066526" 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "instance": "10.142.0.24:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "instance": "10.142.0.74:9100" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "6260.533333333333" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066526" 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "ds-idrepo-1" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "ds-idrepo-0" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG }, 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG "value": [ 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:26 DEBUG 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG [...] 18:35:26 DEBUG { 18:35:26 DEBUG 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG 18:35:26 DEBUG "data": { 18:35:26 DEBUG 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:35:26 DEBUG "result": [ 18:35:26 DEBUG }, 18:35:26 DEBUG { 18:35:26 DEBUG "value": [ 18:35:26 DEBUG "metric": { 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:35:26 DEBUG "351506432" 18:35:26 DEBUG }, 18:35:26 DEBUG ] 18:35:26 DEBUG "value": [ 18:35:26 DEBUG } 18:35:26 DEBUG 18:35:26 DEBUG ] 18:35:26 DEBUG 18:35:26 DEBUG } 18:35:26 DEBUG 18:35:26 DEBUG } 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "ds-cts-0" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "ds-cts-2" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066526" 18:35:26 INFO [http_cmd]: http status code OK 18:35:26 DEBUG --- status code --- 18:35:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:26 DEBUG --- http response --- 18:35:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:35:26 DEBUG ----- output ----- 18:35:26 DEBUG { 18:35:26 DEBUG "status": "success", 18:35:26 DEBUG "data": { 18:35:26 DEBUG "resultType": "vector", 18:35:26 DEBUG "result": [ 18:35:26 DEBUG { 18:35:26 DEBUG "metric": { 18:35:26 DEBUG "pod": "ds-cts-0" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG [...] 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG 18:35:26 DEBUG "pod": "ds-cts-2" 18:35:26 DEBUG }, 18:35:26 DEBUG "value": [ 18:35:26 DEBUG 1714066526, 18:35:26 DEBUG "0" 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG ] 18:35:26 DEBUG } 18:35:26 DEBUG } 18:35:26 INFO 18:35:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:35:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:35:26 INFO [loop_until]: OK (rc = 0) 18:35:26 DEBUG --- stdout --- 18:35:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 153m 3372Mi am-769b98f8d5-phmxx 115m 3625Mi am-769b98f8d5-zbtk2 125m 3362Mi ds-cts-0 11m 501Mi ds-cts-1 11m 484Mi ds-cts-2 10m 499Mi ds-idrepo-0 4251m 3570Mi ds-idrepo-1 4016m 3556Mi ds-idrepo-2 4032m 3599Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 14m 1102Mi idm-6c95d7588f-n2wtz 12m 1063Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 194m 243Mi 18:35:26 DEBUG --- stderr --- 18:35:26 DEBUG 18:35:26 INFO 18:35:26 INFO [loop_until]: kubectl --namespace=xlou top node 18:35:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:35:26 INFO [loop_until]: OK (rc = 0) 18:35:26 DEBUG --- stdout --- 18:35:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3936m 24% 5822Mi 9% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3749m 23% 4771Mi 8% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4257m 26% 4790Mi 8% gke-xlou-cdm-default-pool-9d1e5395-qd5b 63m 0% 1793Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 175m 1% 5991Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 60m 0% 1676Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 65m 0% 1654Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 204m 1% 5010Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 71m 0% 2940Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 181m 1% 3655Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 164m 1% 5189Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 65m 0% 1397Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 73m 0% 2693Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 263m 1% 3783Mi 6% 18:35:26 DEBUG --- stderr --- 18:35:26 DEBUG 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066586" 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- http response --- 18:36:26 ERROR [http_cmd]: ERROR 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:26 ERROR --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG ----- output ----- 18:36:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG { 18:36:26 ERROR --- http response --- 18:36:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:36:26 DEBUG "data": { 18:36:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:36:26 DEBUG "resultType": "vector", 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG "result": [ 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG { 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG "metric": { 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG "instance": "10.142.0.24:9100" 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:36:26 DEBUG }, 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG "value": [ 18:36:26 DEBUG { 18:36:26 DEBUG { 18:36:26 DEBUG 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG 18:36:26 DEBUG "data": { 18:36:26 DEBUG "data": { 18:36:26 DEBUG 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG [...] 18:36:26 DEBUG "result": [ 18:36:26 DEBUG "result": [ 18:36:26 DEBUG 18:36:26 DEBUG { 18:36:26 DEBUG { 18:36:26 DEBUG 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG 18:36:26 DEBUG "pod": "ds-cts-0" 18:36:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:36:26 DEBUG "instance": "10.142.0.74:9100" 18:36:26 DEBUG }, 18:36:26 DEBUG }, 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 1714066586, 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "10044.266666666665" 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG ] 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG [...] 18:36:26 DEBUG [...] 18:36:26 DEBUG } 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG ] 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG } 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG } 18:36:26 DEBUG { 18:36:26 DEBUG "pod": "ds-cts-2" 18:36:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG }, 18:36:26 DEBUG }, 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:26 DEBUG "data": { 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "value": [ 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG { 18:36:26 DEBUG "result": [ 18:36:26 DEBUG "0" 18:36:26 DEBUG "0.00894086691162727" 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG { 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG "data": { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG "result": [ 18:36:26 DEBUG }, 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG { 18:36:26 DEBUG "value": [ 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG "metric": { 18:36:26 DEBUG 18:36:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:36:26 DEBUG 18:36:26 DEBUG }, 18:36:26 DEBUG 18:36:26 DEBUG "value": [ 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:36:26 DEBUG 18:36:26 DEBUG }, 18:36:26 DEBUG 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:36:26 DEBUG "0" 18:36:26 DEBUG }, 18:36:26 DEBUG ] 18:36:26 DEBUG "value": [ 18:36:26 DEBUG } 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG ] 18:36:26 DEBUG "0.213634549892593" 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066586" 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG { 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG { 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "data": { 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:26 DEBUG "data": { 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG { 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "data": { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "data": { 18:36:26 DEBUG "data": { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "ds-idrepo-1" 18:36:26 DEBUG "instance": "10.142.0.24:9100" 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "pod": "ds-idrepo-1" 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "pod": "ds-idrepo-1" 18:36:26 DEBUG }, 18:36:26 DEBUG }, 18:36:26 DEBUG "result": [ 18:36:26 DEBUG }, 18:36:26 DEBUG "result": [ 18:36:26 DEBUG "result": [ 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "value": [ 18:36:26 DEBUG { 18:36:26 DEBUG "value": [ 18:36:26 DEBUG { 18:36:26 DEBUG { 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "metric": { 18:36:26 DEBUG 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "instance": "10.142.0.24:9100" 18:36:26 DEBUG 18:36:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:36:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG }, 18:36:26 DEBUG 18:36:26 DEBUG }, 18:36:26 DEBUG }, 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG [...] 18:36:26 DEBUG "value": [ 18:36:26 DEBUG [...] 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "value": [ 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "ds-idrepo-0" 18:36:26 DEBUG "instance": "10.142.0.74:9100" 18:36:26 DEBUG [...] 18:36:26 DEBUG "pod": "ds-cts-2" 18:36:26 DEBUG [...] 18:36:26 DEBUG [...] 18:36:26 DEBUG "pod": "ds-idrepo-0" 18:36:26 DEBUG }, 18:36:26 DEBUG }, 18:36:26 DEBUG 18:36:26 DEBUG }, 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG "0" 18:36:26 DEBUG "instance": "10.142.0.74:9100" 18:36:26 DEBUG "0" 18:36:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:36:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG }, 18:36:26 DEBUG ] 18:36:26 DEBUG }, 18:36:26 DEBUG }, 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066586" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG "value": [ 18:36:26 DEBUG } 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "value": [ 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG ] 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG "0.00010000000000000379" 18:36:26 DEBUG } 18:36:26 DEBUG "0" 18:36:26 DEBUG "9310.830167528182" 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "instance": "10.142.0.24:9100" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "instance": "10.142.0.74:9100" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "4518083584" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "510.182474933051" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:36:26 DEBUG }, 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG { 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG "data": { 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "result": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG { 18:36:26 DEBUG 18:36:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:36:26 DEBUG }, 18:36:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 DEBUG 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG [...] 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG 18:36:26 DEBUG } 18:36:26 DEBUG 18:36:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG } 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG ] 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG } 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG } 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "352567296" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:26 DEBUG { 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG { 18:36:26 DEBUG "data": { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "result": [ 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG { 18:36:26 DEBUG "result": [ 18:36:26 DEBUG "metric": { 18:36:26 DEBUG { 18:36:26 DEBUG "pod": "ds-cts-0" 18:36:26 DEBUG "metric": { 18:36:26 DEBUG }, 18:36:26 DEBUG "instance": "10.142.0.24:9100" 18:36:26 DEBUG "value": [ 18:36:26 DEBUG }, 18:36:26 DEBUG 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "ds-cts-2" 18:36:26 DEBUG 18:36:26 DEBUG }, 18:36:26 DEBUG "instance": "10.142.0.74:9100" 18:36:26 DEBUG "value": [ 18:36:26 DEBUG }, 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG "0" 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG ] 18:36:26 DEBUG "3748.533333333333" 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "ds-idrepo-1" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "ds-idrepo-0" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066586" 18:36:26 INFO [http_cmd]: http status code OK 18:36:26 DEBUG --- status code --- 18:36:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:26 DEBUG --- http response --- 18:36:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:36:26 DEBUG ----- output ----- 18:36:26 DEBUG { 18:36:26 DEBUG "status": "success", 18:36:26 DEBUG "data": { 18:36:26 DEBUG "resultType": "vector", 18:36:26 DEBUG "result": [ 18:36:26 DEBUG { 18:36:26 DEBUG "metric": { 18:36:26 DEBUG "pod": "ds-cts-0" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG [...] 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG 18:36:26 DEBUG "pod": "ds-cts-2" 18:36:26 DEBUG }, 18:36:26 DEBUG "value": [ 18:36:26 DEBUG 1714066586, 18:36:26 DEBUG "0" 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG ] 18:36:26 DEBUG } 18:36:26 DEBUG } 18:36:26 INFO 18:36:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:36:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:36:26 INFO [loop_until]: OK (rc = 0) 18:36:26 DEBUG --- stdout --- 18:36:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 141m 3372Mi am-769b98f8d5-phmxx 110m 3626Mi am-769b98f8d5-zbtk2 115m 3362Mi ds-cts-0 9m 501Mi ds-cts-1 10m 484Mi ds-cts-2 10m 499Mi ds-idrepo-0 4354m 3873Mi ds-idrepo-1 3933m 3981Mi ds-idrepo-2 4058m 3776Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 5m 1102Mi idm-6c95d7588f-n2wtz 6m 1063Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 187m 243Mi 18:36:26 DEBUG --- stderr --- 18:36:26 DEBUG 18:36:26 INFO 18:36:26 INFO [loop_until]: kubectl --namespace=xlou top node 18:36:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:36:26 INFO [loop_until]: OK (rc = 0) 18:36:26 DEBUG --- stdout --- 18:36:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 3958m 24% 5955Mi 10% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3989m 25% 5206Mi 8% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4282m 26% 5098Mi 8% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1815Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 174m 1% 6012Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 60m 0% 1693Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 61m 0% 1651Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 191m 1% 5005Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 68m 0% 2935Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 164m 1% 3648Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 167m 1% 5188Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 68m 0% 1389Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 69m 0% 2695Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 257m 1% 3783Mi 6% 18:36:26 DEBUG --- stderr --- 18:36:26 DEBUG 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066646" 18:37:26 ERROR [http_cmd]: ERROR 18:37:26 ERROR --- status code --- 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- status code --- 18:37:26 ERROR --- http response --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:37:26 DEBUG --- http response --- 18:37:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "instance": "10.142.0.24:9100" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "instance": "10.142.0.74:9100" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "10714.633333333331" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0.00866949694147708" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "ds-cts-0" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "ds-cts-2" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0.06666666666666667" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG }, 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG "value": [ 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG { 18:37:26 DEBUG "0" 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG ] 18:37:26 DEBUG "data": { 18:37:26 DEBUG } 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG ] 18:37:26 DEBUG "result": [ 18:37:26 DEBUG } 18:37:26 DEBUG { 18:37:26 DEBUG } 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0.20470165092444276" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "ds-idrepo-1" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "ds-cts-2" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- status code --- 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG { 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "data": { 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "result": [ 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG "data": { 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG { 18:37:26 DEBUG "resultType": "vector", 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG "result": [ 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "result": [ 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG { 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:37:26 DEBUG }, 18:37:26 DEBUG "pod": "ds-idrepo-1" 18:37:26 DEBUG { 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG "metric": { 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "data": { 18:37:26 DEBUG { 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG }, 18:37:26 DEBUG "metric": { 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "ds-idrepo-1" 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG "value": [ 18:37:26 DEBUG "instance": "10.142.0.24:9100" 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG }, 18:37:26 DEBUG "instance": "10.142.0.24:9100" 18:37:26 DEBUG }, 18:37:26 DEBUG "result": [ 18:37:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG }, 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG "value": [ 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG { 18:37:26 DEBUG }, 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "value": [ 18:37:26 DEBUG { 18:37:26 DEBUG 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "instance": "10.142.0.24:9100" 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG "data": { 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG }, 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG "result": [ 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG { 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:37:26 DEBUG }, 18:37:26 DEBUG "pod": "ds-idrepo-0" 18:37:26 DEBUG 18:37:26 DEBUG "metric": { 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG }, 18:37:26 DEBUG 18:37:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:37:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:37:26 DEBUG 18:37:26 DEBUG "pod": "ds-idrepo-0" 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG "instance": "10.142.0.74:9100" 18:37:26 DEBUG }, 18:37:26 DEBUG }, 18:37:26 DEBUG "instance": "10.142.0.74:9100" 18:37:26 DEBUG }, 18:37:26 DEBUG 18:37:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG "value": [ 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG }, 18:37:26 DEBUG "0" 18:37:26 DEBUG ] 18:37:26 DEBUG "0" 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG 18:37:26 DEBUG "value": [ 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG 18:37:26 DEBUG "0" 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG "instance": "10.142.0.74:9100" 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG "0" 18:37:26 DEBUG 18:37:26 DEBUG ] 18:37:26 DEBUG "0.0002333333333332632" 18:37:26 DEBUG ] 18:37:26 DEBUG }, 18:37:26 DEBUG "34058915.0004565" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG ] 18:37:26 DEBUG [...] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG "value": [ 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG "4518304768" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG }, 18:37:26 DEBUG ] 18:37:26 DEBUG "value": [ 18:37:26 DEBUG } 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG } 18:37:26 DEBUG "498.61529565720195" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:37:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:37:26 DEBUG }, 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:37:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:37:26 DEBUG }, 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG "0" 18:37:26 DEBUG ] 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "352600064" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "ds-cts-0" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG [...] 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG 18:37:26 DEBUG { 18:37:26 DEBUG "pod": "ds-cts-2" 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG }, 18:37:26 DEBUG "data": { 18:37:26 DEBUG "value": [ 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "result": [ 18:37:26 DEBUG "0" 18:37:26 DEBUG { 18:37:26 DEBUG ] 18:37:26 DEBUG "metric": { 18:37:26 DEBUG } 18:37:26 DEBUG "instance": "10.142.0.24:9100" 18:37:26 DEBUG ] 18:37:26 DEBUG }, 18:37:26 DEBUG } 18:37:26 DEBUG "value": [ 18:37:26 DEBUG } 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "instance": "10.142.0.74:9100" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "4470.733333333333" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "ds-idrepo-1" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "ds-idrepo-0" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066646" 18:37:26 INFO [http_cmd]: http status code OK 18:37:26 DEBUG --- status code --- 18:37:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:26 DEBUG --- http response --- 18:37:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:37:26 DEBUG ----- output ----- 18:37:26 DEBUG { 18:37:26 DEBUG "status": "success", 18:37:26 DEBUG "data": { 18:37:26 DEBUG "resultType": "vector", 18:37:26 DEBUG "result": [ 18:37:26 DEBUG { 18:37:26 DEBUG "metric": { 18:37:26 DEBUG "pod": "ds-cts-0" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG [...] 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG 18:37:26 DEBUG "pod": "ds-cts-2" 18:37:26 DEBUG }, 18:37:26 DEBUG "value": [ 18:37:26 DEBUG 1714066646, 18:37:26 DEBUG "0" 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG ] 18:37:26 DEBUG } 18:37:26 DEBUG } 18:37:26 INFO 18:37:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:37:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:37:26 INFO [loop_until]: OK (rc = 0) 18:37:26 DEBUG --- stdout --- 18:37:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 133m 3372Mi am-769b98f8d5-phmxx 112m 3625Mi am-769b98f8d5-zbtk2 115m 3362Mi ds-cts-0 9m 502Mi ds-cts-1 9m 484Mi ds-cts-2 7m 500Mi ds-idrepo-0 4320m 4094Mi ds-idrepo-1 4095m 4442Mi ds-idrepo-2 4121m 4217Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 8m 1096Mi idm-6c95d7588f-n2wtz 8m 1063Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 185m 244Mi 18:37:26 DEBUG --- stderr --- 18:37:26 DEBUG 18:37:26 INFO 18:37:26 INFO [loop_until]: kubectl --namespace=xlou top node 18:37:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:37:26 INFO [loop_until]: OK (rc = 0) 18:37:26 DEBUG --- stdout --- 18:37:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 4110m 25% 6440Mi 10% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3939m 24% 5489Mi 9% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4399m 27% 5317Mi 9% gke-xlou-cdm-default-pool-9d1e5395-qd5b 60m 0% 1815Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 172m 1% 6007Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 58m 0% 1688Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 64m 0% 1654Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 181m 1% 5005Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 71m 0% 2943Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 165m 1% 3637Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 163m 1% 5193Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 65m 0% 1406Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 62m 0% 2698Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 264m 1% 3782Mi 6% 18:37:26 DEBUG --- stderr --- 18:37:26 DEBUG 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 ERROR [http_cmd]: ERROR 18:38:26 ERROR --- status code --- 18:38:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 ERROR --- http response --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:38:26 DEBUG --- http response --- 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG { 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG "data": { 18:38:26 DEBUG { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "data": { 18:38:26 DEBUG { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "instance": "10.142.0.24:9100" 18:38:26 DEBUG { 18:38:26 DEBUG }, 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "instance": "10.142.0.74:9100" 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG }, 18:38:26 DEBUG "10472.566666666666" 18:38:26 DEBUG "value": [ 18:38:26 DEBUG ] 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG } 18:38:26 DEBUG "0.20799428478854626" 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "ds-cts-0" 18:38:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "ds-cts-2" 18:38:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0" 18:38:26 DEBUG "0.17851625715031152" 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "ds-idrepo-1" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "ds-cts-2" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:38:26 DEBUG "pod": "ds-idrepo-1" 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:38:26 DEBUG "pod": "ds-idrepo-0" 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "2538.0425920490493" 18:38:26 DEBUG "0" 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 DEBUG --- status code --- 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG --- http response --- 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG { 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG "data": { 18:38:26 DEBUG { 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "data": { 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG { 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "data": { 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "result": [ 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG "data": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG { 18:38:26 DEBUG "result": [ 18:38:26 DEBUG }, 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "data": { 18:38:26 DEBUG "data": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:38:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG { 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "pod": "ds-idrepo-1" 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "instance": "10.142.0.24:9100" 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG "data": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "result": [ 18:38:26 DEBUG }, 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG { 18:38:26 DEBUG { 18:38:26 DEBUG "value": [ 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "result": [ 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG { 18:38:26 DEBUG "value": [ 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG "instance": "10.142.0.24:9100" 18:38:26 DEBUG "instance": "10.142.0.24:9100" 18:38:26 DEBUG 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "metric": { 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG [...] 18:38:26 DEBUG [...] 18:38:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG [...] 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG { 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG [...] 18:38:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "data": { 18:38:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:38:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "pod": "ds-idrepo-0" 18:38:26 DEBUG [...] 18:38:26 DEBUG [...] 18:38:26 DEBUG "instance": "10.142.0.74:9100" 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG [...] 18:38:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:38:26 DEBUG "0" 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG { 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG ] 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "0" 18:38:26 DEBUG "0" 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG } 18:38:26 DEBUG "0" 18:38:26 DEBUG "instance": "10.142.0.74:9100" 18:38:26 DEBUG "instance": "10.142.0.74:9100" 18:38:26 DEBUG "0.00033333333333333365" 18:38:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG }, 18:38:26 DEBUG }, 18:38:26 DEBUG ] 18:38:26 DEBUG }, 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:38:26 DEBUG "0" 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "value": [ 18:38:26 DEBUG } 18:38:26 DEBUG "value": [ 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG }, 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG ] 18:38:26 DEBUG 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG "value": [ 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG "0.00020000000000000757" 18:38:26 DEBUG "4518558720" 18:38:26 DEBUG } 18:38:26 DEBUG 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG 18:38:26 DEBUG "0" 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG [...] 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG ] 18:38:26 DEBUG 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 DEBUG 18:38:26 DEBUG } 18:38:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:38:26 DEBUG } 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0.0005" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:38:26 DEBUG }, 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "351719424" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG { 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:38:26 DEBUG "data": { 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG { 18:38:26 DEBUG "result": [ 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG { 18:38:26 DEBUG "data": { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "instance": "10.142.0.24:9100" 18:38:26 DEBUG "result": [ 18:38:26 DEBUG }, 18:38:26 DEBUG { 18:38:26 DEBUG "value": [ 18:38:26 DEBUG "metric": { 18:38:26 DEBUG 18:38:26 DEBUG "pod": "ds-cts-0" 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG "instance": "10.142.0.74:9100" 18:38:26 DEBUG 18:38:26 DEBUG }, 18:38:26 DEBUG 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "pod": "ds-cts-2" 18:38:26 DEBUG "4232.166666666667" 18:38:26 DEBUG }, 18:38:26 DEBUG ] 18:38:26 DEBUG "value": [ 18:38:26 DEBUG } 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG ] 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG "0" 18:38:26 DEBUG } 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG } 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG ] 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:26 DEBUG } 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG } 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "ds-idrepo-1" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "ds-idrepo-0" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066706" 18:38:26 INFO [http_cmd]: http status code OK 18:38:26 DEBUG --- status code --- 18:38:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:26 DEBUG --- http response --- 18:38:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:38:26 DEBUG ----- output ----- 18:38:26 DEBUG { 18:38:26 DEBUG "status": "success", 18:38:26 DEBUG "data": { 18:38:26 DEBUG "resultType": "vector", 18:38:26 DEBUG "result": [ 18:38:26 DEBUG { 18:38:26 DEBUG "metric": { 18:38:26 DEBUG "pod": "ds-cts-0" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG [...] 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG 18:38:26 DEBUG "pod": "ds-cts-2" 18:38:26 DEBUG }, 18:38:26 DEBUG "value": [ 18:38:26 DEBUG 1714066706, 18:38:26 DEBUG "0" 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG ] 18:38:26 DEBUG } 18:38:26 DEBUG } 18:38:26 INFO 18:38:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:38:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:38:26 INFO [loop_until]: OK (rc = 0) 18:38:26 DEBUG --- stdout --- 18:38:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 133m 3372Mi am-769b98f8d5-phmxx 100m 3626Mi am-769b98f8d5-zbtk2 111m 3362Mi ds-cts-0 9m 502Mi ds-cts-1 10m 485Mi ds-cts-2 7m 500Mi ds-idrepo-0 4365m 4441Mi ds-idrepo-1 3987m 4433Mi ds-idrepo-2 4045m 4256Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 5m 1096Mi idm-6c95d7588f-n2wtz 5m 1063Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 174m 244Mi 18:38:26 DEBUG --- stderr --- 18:38:26 DEBUG 18:38:27 INFO 18:38:27 INFO [loop_until]: kubectl --namespace=xlou top node 18:38:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:38:27 INFO [loop_until]: OK (rc = 0) 18:38:27 DEBUG --- stdout --- 18:38:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 4083m 25% 6465Mi 11% gke-xlou-cdm-default-pool-9d1e5395-4r9x 3984m 25% 5740Mi 9% gke-xlou-cdm-default-pool-9d1e5395-jnd9 4367m 27% 5680Mi 9% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1798Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 170m 1% 5986Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 64m 0% 1692Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 64m 0% 1650Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 185m 1% 5006Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 72m 0% 2947Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 163m 1% 3655Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 153m 0% 5194Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 66m 0% 1400Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 63m 0% 2687Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 245m 1% 3784Mi 6% 18:38:27 DEBUG --- stderr --- 18:38:27 DEBUG 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 ERROR [http_cmd]: ERROR 18:39:26 ERROR --- status code --- 18:39:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:39:26 ERROR --- http response --- 18:39:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:39:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "data": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "0.6708017444488803" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "data": { 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG "data": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "pod": "ds-idrepo-1" 18:39:26 DEBUG "data": { 18:39:26 DEBUG "instance": "10.142.0.24:9100" 18:39:26 DEBUG }, 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG { 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "metric": { 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "ds-cts-0" 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG }, 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "ds-cts-2" 18:39:26 DEBUG 18:39:26 DEBUG "instance": "10.142.0.74:9100" 18:39:26 DEBUG }, 18:39:26 DEBUG [...] 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "0" 18:39:26 DEBUG 18:39:26 DEBUG "10342.3" 18:39:26 DEBUG ] 18:39:26 DEBUG "pod": "ds-cts-2" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG }, 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG "value": [ 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 DEBUG } 18:39:26 DEBUG "0" 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "data": { 18:39:26 DEBUG { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "data": { 18:39:26 DEBUG { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:39:26 DEBUG { 18:39:26 DEBUG }, 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG }, 18:39:26 DEBUG "0" 18:39:26 DEBUG "value": [ 18:39:26 DEBUG ] 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG } 18:39:26 DEBUG "0.8599068993340344" 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:39:26 DEBUG { 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG { 18:39:26 DEBUG "data": { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "data": { 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG { 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "metric": { 18:39:26 DEBUG { 18:39:26 DEBUG "pod": "ds-idrepo-1" 18:39:26 DEBUG "metric": { 18:39:26 DEBUG }, 18:39:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:39:26 DEBUG "value": [ 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "ds-idrepo-0" 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:39:26 DEBUG "value": [ 18:39:26 DEBUG }, 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "0" 18:39:26 DEBUG "value": [ 18:39:26 DEBUG ] 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG } 18:39:26 DEBUG "33931097.05892745" 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "data": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "132.4272835611637" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066766" 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "data": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "4121542656" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "data": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "pod": "ds-cts-0" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "ds-cts-2" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "0" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG "data": { 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG "result": [ 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG { 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG "metric": { 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "instance": "10.142.0.24:9100" 18:39:26 DEBUG { 18:39:26 DEBUG }, 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "data": { 18:39:26 DEBUG 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG 18:39:26 DEBUG "result": [ 18:39:26 DEBUG 18:39:26 DEBUG { 18:39:26 DEBUG [...] 18:39:26 DEBUG "metric": { 18:39:26 DEBUG 18:39:26 DEBUG "instance": "10.142.0.24:9100" 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "instance": "10.142.0.74:9100" 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG [...] 18:39:26 DEBUG "4368.533333333333" 18:39:26 DEBUG 18:39:26 DEBUG ] 18:39:26 DEBUG 18:39:26 DEBUG } 18:39:26 DEBUG 18:39:26 DEBUG ] 18:39:26 DEBUG "instance": "10.142.0.74:9100" 18:39:26 DEBUG } 18:39:26 DEBUG }, 18:39:26 DEBUG } 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "0" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- status code --- 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG --- http response --- 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG { 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG { 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "data": { 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG { 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "data": { 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG "data": { 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "data": { 18:39:26 DEBUG "data": { 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "data": { 18:39:26 DEBUG "data": { 18:39:26 DEBUG "data": { 18:39:26 DEBUG { 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "metric": { 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:26 DEBUG { 18:39:26 DEBUG "data": { 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG }, 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "pod": "ds-idrepo-1" 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "value": [ 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG { 18:39:26 DEBUG }, 18:39:26 DEBUG { 18:39:26 DEBUG "pod": "ds-idrepo-1" 18:39:26 DEBUG "result": [ 18:39:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:39:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:39:26 DEBUG 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG }, 18:39:26 DEBUG { 18:39:26 DEBUG }, 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "instance": "10.142.0.24:9100" 18:39:26 DEBUG "instance": "10.142.0.24:9100" 18:39:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:39:26 DEBUG 18:39:26 DEBUG "data": { 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG }, 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG 18:39:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG "result": [ 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG { 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "metric": { 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:39:26 DEBUG [...] 18:39:26 DEBUG [...] 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "ds-idrepo-0" 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG "pod": "ds-idrepo-0" 18:39:26 DEBUG 18:39:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:39:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG 18:39:26 DEBUG }, 18:39:26 DEBUG }, 18:39:26 DEBUG "0" 18:39:26 DEBUG "instance": "10.142.0.74:9100" 18:39:26 DEBUG "instance": "10.142.0.74:9100" 18:39:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "value": [ 18:39:26 DEBUG ] 18:39:26 DEBUG }, 18:39:26 DEBUG }, 18:39:26 DEBUG }, 18:39:26 DEBUG "0" 18:39:26 DEBUG [...] 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG } 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "value": [ 18:39:26 DEBUG "value": [ 18:39:26 DEBUG ] 18:39:26 DEBUG 18:39:26 DEBUG "0" 18:39:26 DEBUG }, 18:39:26 DEBUG "0" 18:39:26 DEBUG "0" 18:39:26 DEBUG ] 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG } 18:39:26 DEBUG 18:39:26 DEBUG ] 18:39:26 DEBUG "value": [ 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG "0.00010000000000000379" 18:39:26 DEBUG "4518804480" 18:39:26 DEBUG "0" 18:39:26 DEBUG ] 18:39:26 DEBUG 18:39:26 DEBUG } 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:39:26 DEBUG ] 18:39:26 DEBUG "0" 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG }, 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG ] 18:39:26 DEBUG "value": [ 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG "0.001" 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "data": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "0" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066766" 18:39:26 INFO [http_cmd]: http status code OK 18:39:26 DEBUG --- status code --- 18:39:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:26 DEBUG --- http response --- 18:39:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:39:26 DEBUG ----- output ----- 18:39:26 DEBUG { 18:39:26 DEBUG "status": "success", 18:39:26 DEBUG "data": { 18:39:26 DEBUG "resultType": "vector", 18:39:26 DEBUG "result": [ 18:39:26 DEBUG { 18:39:26 DEBUG "metric": { 18:39:26 DEBUG "pod": "ds-cts-0" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG [...] 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG 18:39:26 DEBUG "pod": "ds-cts-2" 18:39:26 DEBUG }, 18:39:26 DEBUG "value": [ 18:39:26 DEBUG 1714066766, 18:39:26 DEBUG "0" 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG ] 18:39:26 DEBUG } 18:39:26 DEBUG } 18:39:26 INFO 18:39:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:39:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:39:26 INFO [loop_until]: OK (rc = 0) 18:39:26 DEBUG --- stdout --- 18:39:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 5m 3372Mi am-769b98f8d5-phmxx 5m 3626Mi am-769b98f8d5-zbtk2 5m 3362Mi ds-cts-0 8m 503Mi ds-cts-1 7m 485Mi ds-cts-2 7m 500Mi ds-idrepo-0 42m 4431Mi ds-idrepo-1 47m 4509Mi ds-idrepo-2 56m 4226Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 7m 1096Mi idm-6c95d7588f-n2wtz 10m 1064Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 864m 1013Mi 18:39:26 DEBUG --- stderr --- 18:39:26 DEBUG 18:39:27 INFO 18:39:27 INFO [loop_until]: kubectl --namespace=xlou top node 18:39:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:39:27 INFO [loop_until]: OK (rc = 0) 18:39:27 DEBUG --- stdout --- 18:39:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 102m 0% 6452Mi 11% gke-xlou-cdm-default-pool-9d1e5395-4r9x 90m 0% 5738Mi 9% gke-xlou-cdm-default-pool-9d1e5395-jnd9 73m 0% 5685Mi 9% gke-xlou-cdm-default-pool-9d1e5395-qd5b 60m 0% 1810Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 67m 0% 5991Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 60m 0% 1705Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 58m 0% 1654Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 68m 0% 4988Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 75m 0% 2974Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 182m 1% 3669Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 66m 0% 5186Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 64m 0% 1388Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 65m 0% 2695Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 864m 5% 4553Mi 7% 18:39:27 DEBUG --- stderr --- 18:39:27 DEBUG 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066826" 18:40:26 ERROR [http_cmd]: ERROR 18:40:26 ERROR --- status code --- 18:40:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:40:26 ERROR --- http response --- 18:40:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:40:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "0.24912028839793277" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "0" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "pod": "ds-cts-0" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "ds-cts-2" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "0" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "0.32949598815499537" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG "result": [ 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG { 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG "metric": { 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG "pod": "ds-idrepo-1" 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:26 DEBUG }, 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG "value": [ 18:40:26 DEBUG { 18:40:26 DEBUG 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG 18:40:26 DEBUG "data": { 18:40:26 DEBUG 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG [...] 18:40:26 DEBUG "result": [ 18:40:26 DEBUG 18:40:26 DEBUG { 18:40:26 DEBUG 18:40:26 DEBUG "metric": { 18:40:26 DEBUG 18:40:26 DEBUG "instance": "10.142.0.24:9100" 18:40:26 DEBUG "pod": "ds-cts-2" 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 18:40:26 DEBUG "0" 18:40:26 DEBUG 18:40:26 DEBUG ] 18:40:26 DEBUG [...] 18:40:26 DEBUG } 18:40:26 DEBUG 18:40:26 DEBUG ] 18:40:26 DEBUG 18:40:26 DEBUG } 18:40:26 DEBUG 18:40:26 DEBUG } 18:40:26 DEBUG "instance": "10.142.0.74:9100" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "14382.33333333333" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "data": { 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG "result": [ 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "pod": "ds-idrepo-1" 18:40:26 DEBUG "pod": "ds-cts-0" 18:40:26 DEBUG "instance": "10.142.0.24:9100" 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG [...] 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "ds-idrepo-0" 18:40:26 DEBUG "pod": "ds-cts-2" 18:40:26 DEBUG "instance": "10.142.0.74:9100" 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG }, 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG "value": [ 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG "0" 18:40:26 DEBUG "0" 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG "9841.366666666667" 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG "data": { 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:40:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:40:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "63238928.7548817" 18:40:26 DEBUG "2437.1826593943606" 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066826" 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "1160974336" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066826" 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG { 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:26 DEBUG { 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG "metric": { 18:40:26 DEBUG { 18:40:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG }, 18:40:26 DEBUG "data": { 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG 18:40:26 DEBUG "result": [ 18:40:26 DEBUG 18:40:26 DEBUG { 18:40:26 DEBUG 18:40:26 DEBUG "metric": { 18:40:26 DEBUG [...] 18:40:26 DEBUG "instance": "10.142.0.24:9100" 18:40:26 DEBUG 18:40:26 DEBUG }, 18:40:26 DEBUG 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:40:26 DEBUG 18:40:26 DEBUG }, 18:40:26 DEBUG 18:40:26 DEBUG "value": [ 18:40:26 DEBUG [...] 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 18:40:26 DEBUG "0.001" 18:40:26 DEBUG 18:40:26 DEBUG ] 18:40:26 DEBUG 18:40:26 DEBUG } 18:40:26 DEBUG "instance": "10.142.0.74:9100" 18:40:26 DEBUG ] 18:40:26 DEBUG }, 18:40:26 DEBUG } 18:40:26 DEBUG "value": [ 18:40:26 DEBUG } 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "0" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "instance": "10.142.0.24:9100" 18:40:26 DEBUG "instance": "10.142.0.24:9100" 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "instance": "10.142.0.74:9100" 18:40:26 DEBUG "instance": "10.142.0.74:9100" 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "4519095296" 18:40:26 DEBUG "0.00030000000000001136" 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG --- http response --- 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG { 18:40:26 DEBUG ----- output ----- 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG { 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "status": "success", 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG "data": { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG "data": { 18:40:26 DEBUG "data": { 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "data": { 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "result": [ 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:40:26 DEBUG "result": [ 18:40:26 DEBUG "result": [ 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG { 18:40:26 DEBUG "result": [ 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:26 DEBUG "data": { 18:40:26 DEBUG "data": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "pod": "ds-idrepo-1" 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "pod": "ds-idrepo-1" 18:40:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:40:26 DEBUG { 18:40:26 DEBUG "result": [ 18:40:26 DEBUG "result": [ 18:40:26 DEBUG }, 18:40:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:40:26 DEBUG "data": { 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG { 18:40:26 DEBUG { 18:40:26 DEBUG "value": [ 18:40:26 DEBUG }, 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "data": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "result": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:40:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG { 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "result": [ 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "metric": { 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG { 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:40:26 DEBUG [...] 18:40:26 DEBUG [...] 18:40:26 DEBUG "metric": { 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG }, 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG }, 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "value": [ 18:40:26 DEBUG [...] 18:40:26 DEBUG [...] 18:40:26 DEBUG "pod": "ds-idrepo-0" 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "ds-idrepo-0" 18:40:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG }, 18:40:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:40:26 DEBUG 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "value": [ 18:40:26 DEBUG }, 18:40:26 DEBUG [...] 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG [...] 18:40:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:40:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:40:26 DEBUG "0" 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 18:40:26 DEBUG "0" 18:40:26 DEBUG "0" 18:40:26 DEBUG 18:40:26 DEBUG }, 18:40:26 DEBUG }, 18:40:26 DEBUG ] 18:40:26 DEBUG "0" 18:40:26 DEBUG 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG 18:40:26 DEBUG "value": [ 18:40:26 DEBUG "value": [ 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG "pod": "overseer-0-56c5c978d9-v8wm9" 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG }, 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:40:26 DEBUG "0" 18:40:26 DEBUG "0" 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG "value": [ 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG }, 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG "value": [ 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG "0" 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG ] 18:40:26 DEBUG "1" 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066826" 18:40:26 INFO [http_cmd]: http status code OK 18:40:26 DEBUG --- status code --- 18:40:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:26 DEBUG --- http response --- 18:40:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:40:26 DEBUG ----- output ----- 18:40:26 DEBUG { 18:40:26 DEBUG "status": "success", 18:40:26 DEBUG "data": { 18:40:26 DEBUG "resultType": "vector", 18:40:26 DEBUG "result": [ 18:40:26 DEBUG { 18:40:26 DEBUG "metric": { 18:40:26 DEBUG "pod": "ds-cts-0" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG [...] 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG 18:40:26 DEBUG "pod": "ds-cts-2" 18:40:26 DEBUG }, 18:40:26 DEBUG "value": [ 18:40:26 DEBUG 1714066826, 18:40:26 DEBUG "0" 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG ] 18:40:26 DEBUG } 18:40:26 DEBUG } 18:40:26 INFO 18:40:26 INFO [loop_until]: kubectl --namespace=xlou top pods 18:40:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:40:27 INFO [loop_until]: OK (rc = 0) 18:40:27 DEBUG --- stdout --- 18:40:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 7m 3382Mi am-769b98f8d5-phmxx 17m 3645Mi am-769b98f8d5-zbtk2 25m 3389Mi ds-cts-0 453m 503Mi ds-cts-1 421m 487Mi ds-cts-2 327m 501Mi ds-idrepo-0 543m 4481Mi ds-idrepo-1 654m 4559Mi ds-idrepo-2 534m 4245Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 7m 1112Mi idm-6c95d7588f-n2wtz 6m 1070Mi lodemon-596c8449d-bbnvt 2m 62Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 223m 1171Mi 18:40:27 DEBUG --- stderr --- 18:40:27 DEBUG 18:40:27 INFO 18:40:27 INFO [loop_until]: kubectl --namespace=xlou top node 18:40:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:40:27 INFO [loop_until]: OK (rc = 0) 18:40:27 DEBUG --- stdout --- 18:40:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 614m 3% 6418Mi 10% gke-xlou-cdm-default-pool-9d1e5395-4r9x 601m 3% 5741Mi 9% gke-xlou-cdm-default-pool-9d1e5395-jnd9 651m 4% 5702Mi 9% gke-xlou-cdm-default-pool-9d1e5395-qd5b 764m 4% 1818Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 72m 0% 6012Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 437m 2% 1669Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 438m 2% 1611Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 70m 0% 4984Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 71m 0% 2957Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 179m 1% 3667Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 73m 0% 5203Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 69m 0% 1390Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 64m 0% 2700Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 300m 1% 4708Mi 8% 18:40:27 DEBUG --- stderr --- 18:40:27 DEBUG 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066886" 18:41:26 ERROR [http_cmd]: ERROR 18:41:26 ERROR --- status code --- 18:41:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:41:26 ERROR --- http response --- 18:41:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:41:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- http response --- 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG ----- output ----- 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG { 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG "data": { 18:41:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG "result": [ 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "data": { 18:41:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG }, 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG { 18:41:26 DEBUG 18:41:26 DEBUG "metric": { 18:41:26 DEBUG 18:41:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG [...] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:41:26 DEBUG [...] 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG }, 18:41:26 DEBUG 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG "0.009289892124461971" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG { 18:41:26 DEBUG ] 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG }, 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG } 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG "value": [ 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG "data": { 18:41:26 DEBUG ] 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG } 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG "0.3324572044712707" 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:41:26 DEBUG "result": [ 18:41:26 DEBUG } 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG ] 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG } 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG ] 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "instance": "10.142.0.24:9100" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG "data": { 18:41:26 DEBUG } 18:41:26 DEBUG "data": { 18:41:26 DEBUG }, 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG } 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "value": [ 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "result": [ 18:41:26 DEBUG 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:41:26 DEBUG "pod": "ds-cts-0" 18:41:26 DEBUG [...] 18:41:26 DEBUG { 18:41:26 DEBUG }, 18:41:26 DEBUG }, 18:41:26 DEBUG 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG "data": { 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "instance": "10.142.0.74:9100" 18:41:26 DEBUG "result": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG { 18:41:26 DEBUG [...] 18:41:26 DEBUG [...] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "metric": { 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "pod": "ds-idrepo-1" 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "10356.199999999999" 18:41:26 DEBUG }, 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG ] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:41:26 DEBUG "pod": "ds-cts-2" 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG }, 18:41:26 DEBUG ] 18:41:26 DEBUG 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG } 18:41:26 DEBUG [...] 18:41:26 DEBUG "0" 18:41:26 DEBUG "0" 18:41:26 DEBUG 18:41:26 DEBUG ] 18:41:26 DEBUG ] 18:41:26 DEBUG 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG ] 18:41:26 DEBUG ] 18:41:26 DEBUG "pod": "ds-cts-2" 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG }, 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "0" 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "data": { 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "result": [ 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "pod": "ds-cts-0" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG "pod": "ds-cts-2" 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG }, 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG "value": [ 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066886" 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG ----- output ----- 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG "0" 18:41:26 DEBUG { 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG ] 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG } 18:41:26 DEBUG "data": { 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:26 DEBUG "data": { 18:41:26 DEBUG "data": { 18:41:26 DEBUG ] 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG } 18:41:26 DEBUG "result": [ 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "result": [ 18:41:26 DEBUG } 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "data": { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "pod": "ds-idrepo-1" 18:41:26 DEBUG "data": { 18:41:26 DEBUG "data": { 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "instance": "10.142.0.24:9100" 18:41:26 DEBUG "instance": "10.142.0.24:9100" 18:41:26 DEBUG }, 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "result": [ 18:41:26 DEBUG }, 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "result": [ 18:41:26 DEBUG { 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG { 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "instance": "10.142.0.24:9100" 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:41:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:41:26 DEBUG }, 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG }, 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG [...] 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "value": [ 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "pod": "ds-idrepo-0" 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG [...] 18:41:26 DEBUG "instance": "10.142.0.74:9100" 18:41:26 DEBUG "instance": "10.142.0.74:9100" 18:41:26 DEBUG }, 18:41:26 DEBUG [...] 18:41:26 DEBUG [...] 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG { 18:41:26 DEBUG 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "0" 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "instance": "10.142.0.74:9100" 18:41:26 DEBUG "0.00016666666666675193" 18:41:26 DEBUG "4050.933333333333" 18:41:26 DEBUG ] 18:41:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:41:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:41:26 DEBUG "data": { 18:41:26 DEBUG }, 18:41:26 DEBUG ] 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG }, 18:41:26 DEBUG }, 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "value": [ 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "result": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG ] 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG { 18:41:26 DEBUG "4519449600" 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG "3320.3217559393906" 18:41:26 DEBUG "510.8187316829832" 18:41:26 DEBUG "metric": { 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG ] 18:41:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:41:26 DEBUG } 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG }, 18:41:26 DEBUG ] 18:41:26 DEBUG --- status code --- 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG ] 18:41:26 DEBUG ] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG } 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG } 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG [...] 18:41:26 DEBUG { 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG { 18:41:26 DEBUG 18:41:26 DEBUG "data": { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "data": { 18:41:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG }, 18:41:26 DEBUG { 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "metric": { 18:41:26 DEBUG { 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "instance": "10.142.0.24:9100" 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "1162514432" 18:41:26 DEBUG }, 18:41:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:41:26 DEBUG ] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG }, 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG "value": [ 18:41:26 DEBUG ] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG } 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "instance": "10.142.0.74:9100" 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:41:26 DEBUG "value": [ 18:41:26 DEBUG }, 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "0.0003333333333333558" 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG ] 18:41:26 DEBUG "0.001" 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "data": { 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "result": [ 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "0" 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "data": { 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "result": [ 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "0" 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "data": { 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "result": [ 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:41:26 DEBUG }, 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "0" 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "data": { 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "result": [ 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "pod": "ds-idrepo-1" 18:41:26 DEBUG }, 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG "value": [ 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG [...] 18:41:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066886" 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG 18:41:26 DEBUG { 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG { 18:41:26 DEBUG 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG 18:41:26 DEBUG "data": { 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG "data": { 18:41:26 DEBUG "pod": "ds-idrepo-0" 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG }, 18:41:26 DEBUG "result": [ 18:41:26 DEBUG { 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "value": [ 18:41:26 DEBUG { 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG { 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "data": { 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "0" 18:41:26 DEBUG "pod": "ds-idrepo-1" 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:41:26 DEBUG ] 18:41:26 DEBUG }, 18:41:26 DEBUG "result": [ 18:41:26 DEBUG "data": { 18:41:26 DEBUG }, 18:41:26 DEBUG } 18:41:26 DEBUG "value": [ 18:41:26 DEBUG { 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG "value": [ 18:41:26 DEBUG ] 18:41:26 DEBUG 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "result": [ 18:41:26 DEBUG 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:41:26 DEBUG { 18:41:26 DEBUG 18:41:26 DEBUG } 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG "metric": { 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "pod": "ds-idrepo-0" 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:41:26 DEBUG }, 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "0" 18:41:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:41:26 DEBUG 18:41:26 DEBUG "0" 18:41:26 DEBUG ] 18:41:26 DEBUG }, 18:41:26 INFO [http_cmd]: http status code OK 18:41:26 DEBUG 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG "value": [ 18:41:26 DEBUG --- status code --- 18:41:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:26 DEBUG }, 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG "0" 18:41:26 DEBUG --- http response --- 18:41:26 DEBUG "value": [ 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG ----- output ----- 18:41:26 DEBUG "1" 18:41:26 DEBUG ] 18:41:26 DEBUG { 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG "status": "success", 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:26 DEBUG "data": { 18:41:26 DEBUG ] 18:41:26 DEBUG "resultType": "vector", 18:41:26 DEBUG } 18:41:26 DEBUG "result": [ 18:41:26 DEBUG } 18:41:26 DEBUG { 18:41:26 DEBUG "metric": { 18:41:26 DEBUG "pod": "ds-cts-0" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG [...] 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG 18:41:26 DEBUG "pod": "ds-cts-2" 18:41:26 DEBUG }, 18:41:26 DEBUG "value": [ 18:41:26 DEBUG 1714066886, 18:41:26 DEBUG "0" 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG ] 18:41:26 DEBUG } 18:41:26 DEBUG } 18:41:27 INFO 18:41:27 INFO [loop_until]: kubectl --namespace=xlou top pods 18:41:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:41:27 INFO [loop_until]: OK (rc = 0) 18:41:27 DEBUG --- stdout --- 18:41:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 7m 3383Mi am-769b98f8d5-phmxx 7m 3645Mi am-769b98f8d5-zbtk2 7m 3389Mi ds-cts-0 10m 505Mi ds-cts-1 8m 488Mi ds-cts-2 7m 502Mi ds-idrepo-0 106m 4482Mi ds-idrepo-1 16m 4559Mi ds-idrepo-2 112m 4278Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 9m 1113Mi idm-6c95d7588f-n2wtz 8m 1070Mi lodemon-596c8449d-bbnvt 2m 63Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 163m 1066Mi 18:41:27 DEBUG --- stderr --- 18:41:27 DEBUG 18:41:27 INFO 18:41:27 INFO [loop_until]: kubectl --namespace=xlou top node 18:41:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:41:27 INFO [loop_until]: OK (rc = 0) 18:41:27 DEBUG --- stdout --- 18:41:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 74m 0% 6483Mi 11% gke-xlou-cdm-default-pool-9d1e5395-4r9x 55m 0% 5766Mi 9% gke-xlou-cdm-default-pool-9d1e5395-jnd9 74m 0% 5721Mi 9% gke-xlou-cdm-default-pool-9d1e5395-qd5b 61m 0% 1800Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 62m 0% 6007Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 61m 0% 1696Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 60m 0% 1646Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 70m 0% 4990Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 71m 0% 2956Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 175m 1% 3659Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 65m 0% 5205Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 66m 0% 1383Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 63m 0% 2702Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 92m 0% 4556Mi 7% 18:41:27 DEBUG --- stderr --- 18:41:27 DEBUG 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING No values available for selected timeframe. 18:41:55 WARNING File /tmp/lodemon_data-20240425_182547/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [25/Apr/2024 18:41:55] "GET /monitoring/average?start_time=20240425_174309&stop_time=20240425_173954 HTTP/1.1" 200 - 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714066946" 18:42:26 ERROR [http_cmd]: ERROR 18:42:26 ERROR --- status code --- 18:42:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:42:26 ERROR --- http response --- 18:42:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:42:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714066946" 18:42:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "data": { 18:42:26 DEBUG "data": { 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "result": [ 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:42:26 DEBUG "pod": "ds-cts-0" 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG [...] 18:42:26 DEBUG [...] 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:42:26 DEBUG "pod": "ds-cts-2" 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG { 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "data": { 18:42:26 DEBUG "0" 18:42:26 DEBUG "0.03333333333333333" 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG ] 18:42:26 DEBUG ] 18:42:26 DEBUG "result": [ 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG { 18:42:26 DEBUG ] 18:42:26 DEBUG ] 18:42:26 DEBUG "metric": { 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "0.005588800294009299" 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG { 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG "data": { 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "data": { 18:42:26 DEBUG "data": { 18:42:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG }, 18:42:26 DEBUG "result": [ 18:42:26 DEBUG "result": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG 18:42:26 DEBUG "pod": "ds-idrepo-1" 18:42:26 DEBUG "instance": "10.142.0.24:9100" 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG [...] 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:42:26 DEBUG [...] 18:42:26 DEBUG [...] 18:42:26 DEBUG }, 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "0.034972275302558864" 18:42:26 DEBUG "pod": "ds-cts-2" 18:42:26 DEBUG "instance": "10.142.0.74:9100" 18:42:26 DEBUG ] 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG } 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG ] 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG } 18:42:26 DEBUG "0" 18:42:26 DEBUG "12023.466666666665" 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "data": { 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "486.52230797500926" 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG ----- output ----- 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066946" 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG { 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG "data": { 18:42:26 DEBUG "data": { 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG "data": { 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG "result": [ 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066946" 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG "result": [ 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "metric": { 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714066946" 18:42:26 DEBUG "data": { 18:42:26 DEBUG "metric": { 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714066946" 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:42:26 DEBUG "instance": "10.142.0.24:9100" 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "pod": "ds-cts-0" 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG "data": { 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG "result": [ 18:42:26 DEBUG }, 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG { 18:42:26 DEBUG "value": [ 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG "result": [ 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "metric": { 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG { 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG "metric": { 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG "pod": "ds-idrepo-1" 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG [...] 18:42:26 DEBUG [...] 18:42:26 DEBUG "value": [ 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [...] 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG }, 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG 18:42:26 DEBUG "data": { 18:42:26 DEBUG "data": { 18:42:26 DEBUG 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG { 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG { 18:42:26 DEBUG 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG "instance": "10.142.0.74:9100" 18:42:26 DEBUG [...] 18:42:26 DEBUG "pod": "ds-cts-2" 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG 18:42:26 DEBUG "result": [ 18:42:26 DEBUG "result": [ 18:42:26 DEBUG }, 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG }, 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG "data": { 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG "data": { 18:42:26 DEBUG [...] 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG "value": [ 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG { 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG "result": [ 18:42:26 DEBUG 18:42:26 DEBUG "instance": "10.142.0.24:9100" 18:42:26 DEBUG "instance": "10.142.0.24:9100" 18:42:26 DEBUG "0" 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "0.00010000000000000379" 18:42:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:42:26 DEBUG "0" 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG { 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG ] 18:42:26 DEBUG "data": { 18:42:26 DEBUG ] 18:42:26 DEBUG }, 18:42:26 DEBUG ] 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "data": { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "ds-idrepo-0" 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG } 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG } 18:42:26 DEBUG "value": [ 18:42:26 DEBUG } 18:42:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:42:26 DEBUG }, 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG ] 18:42:26 DEBUG "result": [ 18:42:26 DEBUG ] 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG ] 18:42:26 DEBUG }, 18:42:26 DEBUG "result": [ 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG } 18:42:26 DEBUG { 18:42:26 DEBUG } 18:42:26 DEBUG "0.0015" 18:42:26 DEBUG } 18:42:26 DEBUG "value": [ 18:42:26 DEBUG { 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG } 18:42:26 DEBUG "metric": { 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG 18:42:26 DEBUG "metric": { 18:42:26 DEBUG 18:42:26 DEBUG "0" 18:42:26 DEBUG [...] 18:42:26 DEBUG [...] 18:42:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:42:26 DEBUG } 18:42:26 DEBUG 18:42:26 DEBUG "instance": "10.142.0.24:9100" 18:42:26 DEBUG 18:42:26 DEBUG ] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG ] 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG 18:42:26 DEBUG } 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "value": [ 18:42:26 DEBUG } 18:42:26 DEBUG [...] 18:42:26 DEBUG "value": [ 18:42:26 DEBUG [...] 18:42:26 DEBUG ] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG } 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG } 18:42:26 DEBUG "instance": "10.142.0.74:9100" 18:42:26 DEBUG "instance": "10.142.0.74:9100" 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG } 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG [...] 18:42:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:42:26 DEBUG [...] 18:42:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG 18:42:26 DEBUG }, 18:42:26 DEBUG "8222.933333333332" 18:42:26 DEBUG "4519795712" 18:42:26 DEBUG 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG "value": [ 18:42:26 DEBUG ] 18:42:26 DEBUG ] 18:42:26 DEBUG 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:42:26 DEBUG "0" 18:42:26 DEBUG "instance": "10.142.0.74:9100" 18:42:26 DEBUG "4621.961925762588" 18:42:26 DEBUG ] 18:42:26 DEBUG ] 18:42:26 DEBUG }, 18:42:26 DEBUG ] 18:42:26 DEBUG }, 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG "value": [ 18:42:26 DEBUG } 18:42:26 DEBUG "value": [ 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG ] 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG ] 18:42:26 DEBUG "1164218368" 18:42:26 DEBUG } 18:42:26 DEBUG "0" 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "data": { 18:42:26 DEBUG "data": { 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "result": [ 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "ds-cts-0" 18:42:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:42:26 DEBUG }, 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG 18:42:26 DEBUG "pod": "ds-cts-2" 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:42:26 DEBUG }, 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG { 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "0" 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "0" 18:42:26 DEBUG ] 18:42:26 DEBUG "data": { 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG "result": [ 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG { 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 DEBUG "metric": { 18:42:26 DEBUG } 18:42:26 DEBUG "pod": "ds-idrepo-1" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG "pod": "ds-idrepo-0" 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG }, 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG "value": [ 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "0" 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:26 DEBUG ] 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG } 18:42:26 DEBUG { 18:42:26 DEBUG ] 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG } 18:42:26 DEBUG "data": { 18:42:26 DEBUG } 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "pod": "idm-6c95d7588f-n2wtz" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "0" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714066946" 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "data": { 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "ds-idrepo-1" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "pod": "ds-idrepo-0" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "0" 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "data": { 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "0" 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:26 INFO [http_cmd]: http status code OK 18:42:26 DEBUG --- status code --- 18:42:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:26 DEBUG --- http response --- 18:42:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:26 DEBUG ----- output ----- 18:42:26 DEBUG { 18:42:26 DEBUG "status": "success", 18:42:26 DEBUG "data": { 18:42:26 DEBUG "resultType": "vector", 18:42:26 DEBUG "result": [ 18:42:26 DEBUG { 18:42:26 DEBUG "metric": { 18:42:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG [...] 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG 18:42:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:42:26 DEBUG }, 18:42:26 DEBUG "value": [ 18:42:26 DEBUG 1714066946, 18:42:26 DEBUG "1" 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG ] 18:42:26 DEBUG } 18:42:26 DEBUG } 18:42:27 INFO 18:42:27 INFO [loop_until]: kubectl --namespace=xlou top pods 18:42:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:42:27 INFO [loop_until]: OK (rc = 0) 18:42:27 DEBUG --- stdout --- 18:42:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 5Mi am-769b98f8d5-4ftrv 7m 3382Mi am-769b98f8d5-phmxx 9m 3645Mi am-769b98f8d5-zbtk2 6m 3406Mi ds-cts-0 7m 505Mi ds-cts-1 8m 489Mi ds-cts-2 9m 503Mi ds-idrepo-0 12m 4482Mi ds-idrepo-1 25m 4559Mi ds-idrepo-2 12m 4279Mi end-user-ui-58878d7668-l6f8k 1m 3Mi idm-6c95d7588f-lrdb9 5m 1113Mi idm-6c95d7588f-n2wtz 8m 1070Mi lodemon-596c8449d-bbnvt 2m 63Mi login-ui-5fdf49fc66-rzjdb 1m 3Mi overseer-0-56c5c978d9-v8wm9 4m 1014Mi 18:42:27 DEBUG --- stderr --- 18:42:27 DEBUG 18:42:27 INFO 18:42:27 INFO [loop_until]: kubectl --namespace=xlou top node 18:42:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:42:27 INFO [loop_until]: OK (rc = 0) 18:42:27 DEBUG --- stdout --- 18:42:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 65m 0% 6478Mi 11% gke-xlou-cdm-default-pool-9d1e5395-4r9x 63m 0% 5770Mi 9% gke-xlou-cdm-default-pool-9d1e5395-jnd9 64m 0% 5743Mi 9% gke-xlou-cdm-default-pool-9d1e5395-qd5b 58m 0% 1807Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 64m 0% 6019Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 64m 0% 1704Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 58m 0% 1657Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 67m 0% 4989Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 72m 0% 2957Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 181m 1% 3660Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 67m 0% 5198Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 67m 0% 1389Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 64m 0% 2698Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 87m 0% 4594Mi 7% 18:42:27 DEBUG --- stderr --- 18:42:27 DEBUG 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1714067006" 18:43:26 ERROR [http_cmd]: ERROR 18:43:26 ERROR --- status code --- 18:43:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:43:26 ERROR --- http response --- 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:43:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0.4403114009980359" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:43:26 DEBUG { 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "data": { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "data": { 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG { 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "metric": { 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "pod": "ds-cts-0" 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG }, 18:43:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:43:26 DEBUG "pod": "ds-idrepo-1" 18:43:26 DEBUG "value": [ 18:43:26 DEBUG }, 18:43:26 DEBUG }, 18:43:26 DEBUG 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "ds-cts-2" 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG }, 18:43:26 DEBUG "pod": "ds-idrepo-2" 18:43:26 DEBUG "pod": "ds-cts-2" 18:43:26 DEBUG "value": [ 18:43:26 DEBUG }, 18:43:26 DEBUG }, 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "0" 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG ] 18:43:26 DEBUG "0.3026384491870875" 18:43:26 DEBUG "0" 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "instance": "10.142.0.24:9100" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "instance": "10.142.0.74:9100" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "12288.4" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "ds-idrepo-2" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "7480.384195786117" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "ds-idrepo-1" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "ds-idrepo-0" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG { 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG { 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG { 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG "data": { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "data": { 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:43:26 DEBUG "data": { 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "ds-idrepo-1" 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "instance": "10.142.0.24:9100" 18:43:26 DEBUG }, 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "metric": { 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG { 18:43:26 DEBUG "pod": "ds-cts-0" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "data": { 18:43:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "data": { 18:43:26 DEBUG "data": { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG }, 18:43:26 DEBUG "instance": "10.142.0.24:9100" 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG }, 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "result": [ 18:43:26 DEBUG }, 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG { 18:43:26 DEBUG 18:43:26 DEBUG "value": [ 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG "metric": { 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG "instance": "10.142.0.24:9100" 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "instance": "10.142.0.24:9100" 18:43:26 DEBUG "node": "gke-xlou-cdm-ds-174e6abc-k8cj" 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG }, 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG }, 18:43:26 DEBUG }, 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "ds-idrepo-0" 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "instance": "10.142.0.74:9100" 18:43:26 DEBUG }, 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "ds-cts-2" 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG [...] 18:43:26 DEBUG }, 18:43:26 DEBUG "instance": "10.142.0.74:9100" 18:43:26 DEBUG [...] 18:43:26 DEBUG [...] 18:43:26 DEBUG "pod": "ds-idrepo-2" 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG 18:43:26 DEBUG "value": [ 18:43:26 DEBUG }, 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG }, 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0.00036666666666675953" 18:43:26 DEBUG ] 18:43:26 DEBUG 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "0" 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG 18:43:26 DEBUG "0.0035" 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG "instance": "10.142.0.74:9100" 18:43:26 DEBUG ] 18:43:26 DEBUG "4520162304" 18:43:26 DEBUG "instance": "10.142.0.74:9100" 18:43:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:43:26 DEBUG "4259.663222600428" 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG }, 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG }, 18:43:26 DEBUG }, 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG "value": [ 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG "6598.3" 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG "0" 18:43:26 DEBUG "1160331264" 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG "pod": "ds-cts-0" 18:43:26 DEBUG }, 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG [...] 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 DEBUG 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1714067006" 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG 18:43:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "pod": "ds-cts-2" 18:43:26 DEBUG }, 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "admin-ui-c4c7bf584-gwjx6" 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG "pod": "ds-idrepo-1" 18:43:26 DEBUG }, 18:43:26 DEBUG "0" 18:43:26 DEBUG ] 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG } 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "ds-idrepo-2" 18:43:26 DEBUG "pod": "ds-idrepo-0" 18:43:26 DEBUG }, 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG "0" 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 INFO [http_cmd]: http status code OK 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG --- status code --- 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG --- http response --- 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG ----- output ----- 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "status": "success", 18:43:26 DEBUG "data": { 18:43:26 DEBUG "data": { 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "resultType": "vector", 18:43:26 DEBUG "result": [ 18:43:26 DEBUG "result": [ 18:43:26 DEBUG { 18:43:26 DEBUG { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "metric": { 18:43:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:43:26 DEBUG "pod": "am-769b98f8d5-zbtk2" 18:43:26 DEBUG }, 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG [...] 18:43:26 DEBUG [...] 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG 18:43:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:43:26 DEBUG "pod": "am-769b98f8d5-phmxx" 18:43:26 DEBUG }, 18:43:26 DEBUG }, 18:43:26 DEBUG "value": [ 18:43:26 DEBUG "value": [ 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG 1714067006, 18:43:26 DEBUG "0" 18:43:26 DEBUG "1" 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG ] 18:43:26 DEBUG ] 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:26 DEBUG } 18:43:27 INFO 18:43:27 INFO [loop_until]: kubectl --namespace=xlou top pods 18:43:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:43:27 INFO [loop_until]: OK (rc = 0) 18:43:27 DEBUG --- stdout --- 18:43:27 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-gwjx6 1m 7Mi am-769b98f8d5-4ftrv 9m 3396Mi am-769b98f8d5-phmxx 9m 3652Mi am-769b98f8d5-zbtk2 6m 3406Mi ds-cts-0 8m 506Mi ds-cts-1 9m 489Mi ds-cts-2 7m 503Mi ds-idrepo-0 696m 4508Mi ds-idrepo-1 426m 4560Mi ds-idrepo-2 387m 4279Mi end-user-ui-58878d7668-l6f8k 1m 5Mi idm-6c95d7588f-lrdb9 6m 1114Mi idm-6c95d7588f-n2wtz 7m 1074Mi lodemon-596c8449d-bbnvt 2m 63Mi login-ui-5fdf49fc66-rzjdb 1m 4Mi overseer-0-56c5c978d9-v8wm9 4m 1013Mi 18:43:27 DEBUG --- stderr --- 18:43:27 DEBUG 18:43:27 INFO 18:43:27 INFO [loop_until]: kubectl --namespace=xlou top node 18:43:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:43:27 INFO [loop_until]: OK (rc = 0) 18:43:27 DEBUG --- stdout --- 18:43:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-43zw 558m 3% 6458Mi 11% gke-xlou-cdm-default-pool-9d1e5395-4r9x 548m 3% 5729Mi 9% gke-xlou-cdm-default-pool-9d1e5395-jnd9 696m 4% 5728Mi 9% gke-xlou-cdm-default-pool-9d1e5395-qd5b 60m 0% 1805Mi 3% gke-xlou-cdm-default-pool-9d1e5395-sbmc 68m 0% 6031Mi 10% gke-xlou-cdm-default-pool-9d1e5395-sp1w 60m 0% 1693Mi 2% gke-xlou-cdm-default-pool-9d1e5395-v5t6 61m 0% 1644Mi 2% gke-xlou-cdm-ds-174e6abc-czs8 65m 0% 5007Mi 8% gke-xlou-cdm-ds-174e6abc-drzp 71m 0% 2966Mi 5% gke-xlou-cdm-ds-174e6abc-k8cj 159m 1% 3644Mi 6% gke-xlou-cdm-ds-174e6abc-lzjx 65m 0% 5216Mi 8% gke-xlou-cdm-ds-174e6abc-mm5l 65m 0% 1391Mi 2% gke-xlou-cdm-ds-174e6abc-vc5z 62m 0% 2694Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 84m 0% 4637Mi 7% 18:43:27 DEBUG --- stderr --- 18:43:27 DEBUG 18:43:32 INFO Finished: True 18:43:32 INFO Waiting for threads to register finish flag 18:44:27 INFO Done. Have a nice day! :) 127.0.0.1 - - [25/Apr/2024 18:44:27] "GET /monitoring/stop HTTP/1.1" 200 - 18:44:30 WARNING File /tmp/lodemon_data-20240425_182547/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6471', ...) 127.0.0.1 - - [25/Apr/2024 18:44:34] "GET /monitoring/process HTTP/1.1" 200 -