==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-799cdc5bdb-7t5x7 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Sat, 20 Apr 2024 04:32:34 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=799cdc5bdb skaffold.dev/run-id=6255148e-1a42-41c5-8375-68c5f6bacd8e Annotations: Status: Running IP: 10.106.46.169 IPs: IP: 10.106.46.169 Controlled By: ReplicaSet/lodemon-799cdc5bdb Containers: lodemon: Container ID: containerd://53e9fc0c888481158c81e2f9d778666694f35cc71dd0d0603e8c510e6cffb4eb Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sat, 20 Apr 2024 04:32:35 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6w5jt (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-6w5jt: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 41m default-scheduler Successfully assigned xlou/lodemon-799cdc5bdb-7t5x7 to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 41m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 41m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 303.500755ms (303.507999ms including waiting) Normal Created 41m kubelet Created container lodemon Normal Started 41m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.169:8080 Press CTRL+C to quit 05:33:41 INFO 05:33:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:41 INFO [loop_until]: OK (rc = 0) 05:33:41 DEBUG --- stdout --- 05:33:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:41 DEBUG --- stderr --- 05:33:41 DEBUG 05:33:41 INFO 05:33:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:41 INFO [loop_until]: OK (rc = 0) 05:33:41 DEBUG --- stdout --- 05:33:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:41 DEBUG --- stderr --- 05:33:41 DEBUG 05:33:41 INFO 05:33:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:41 INFO [loop_until]: OK (rc = 0) 05:33:41 DEBUG --- stdout --- 05:33:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:41 DEBUG --- stderr --- 05:33:41 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:42 INFO [loop_until]: OK (rc = 0) 05:33:42 DEBUG --- stdout --- 05:33:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:42 DEBUG --- stderr --- 05:33:42 DEBUG 05:33:42 INFO 05:33:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:43 INFO 05:33:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:43 INFO 05:33:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:43 INFO 05:33:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:43 INFO 05:33:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:43 INFO 05:33:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:43 INFO 05:33:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:43 INFO 05:33:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:43 INFO [loop_until]: OK (rc = 0) 05:33:43 DEBUG --- stdout --- 05:33:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:43 DEBUG --- stderr --- 05:33:43 DEBUG 05:33:44 INFO 05:33:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:44 INFO [loop_until]: OK (rc = 0) 05:33:44 DEBUG --- stdout --- 05:33:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:44 DEBUG --- stderr --- 05:33:44 DEBUG 05:33:44 INFO 05:33:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:44 INFO [loop_until]: OK (rc = 0) 05:33:44 DEBUG --- stdout --- 05:33:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:44 DEBUG --- stderr --- 05:33:44 DEBUG 05:33:44 INFO 05:33:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:44 INFO [loop_until]: OK (rc = 0) 05:33:44 DEBUG --- stdout --- 05:33:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:44 DEBUG --- stderr --- 05:33:44 DEBUG 05:33:44 INFO 05:33:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:44 INFO [loop_until]: OK (rc = 0) 05:33:44 DEBUG --- stdout --- 05:33:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:44 DEBUG --- stderr --- 05:33:44 DEBUG 05:33:44 INFO 05:33:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:44 INFO [loop_until]: OK (rc = 0) 05:33:44 DEBUG --- stdout --- 05:33:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:44 DEBUG --- stderr --- 05:33:44 DEBUG 05:33:44 INFO 05:33:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:44 INFO [loop_until]: OK (rc = 0) 05:33:44 DEBUG --- stdout --- 05:33:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:44 DEBUG --- stderr --- 05:33:44 DEBUG 05:33:44 INFO 05:33:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:44 INFO [loop_until]: OK (rc = 0) 05:33:44 DEBUG --- stdout --- 05:33:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:44 DEBUG --- stderr --- 05:33:44 DEBUG 05:33:45 INFO 05:33:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:45 INFO [loop_until]: OK (rc = 0) 05:33:45 DEBUG --- stdout --- 05:33:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:45 DEBUG --- stderr --- 05:33:45 DEBUG 05:33:45 INFO 05:33:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 05:33:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:45 INFO [loop_until]: OK (rc = 0) 05:33:45 DEBUG --- stdout --- 05:33:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 05:33:45 DEBUG --- stderr --- 05:33:45 DEBUG 05:33:45 INFO Initializing monitoring instance threads 05:33:45 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 05:33:45 INFO Starting instance threads 05:33:45 INFO 05:33:45 INFO Thread started 05:33:45 INFO [loop_until]: kubectl --namespace=xlou top node 05:33:45 INFO 05:33:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:45 INFO Thread started 05:33:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:33:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG --- status code --- 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG --- http response --- 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 DEBUG --- status code --- 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 DEBUG --- status code --- 05:33:45 INFO Thread started 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 DEBUG --- http response --- 05:33:45 INFO Thread started 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 INFO [loop_until]: OK (rc = 0) 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG { 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG { 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG --- http response --- 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG --- status code --- 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 DEBUG --- stdout --- 05:33:45 INFO Thread started 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:45 DEBUG { 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG { 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8594Mi am-767f6b67b5-qcqz2 8m 8673Mi am-767f6b67b5-vbltz 7m 8600Mi ds-cts-0 13m 10563Mi ds-cts-1 14m 10672Mi ds-cts-2 14m 10598Mi ds-idrepo-0 13m 3751Mi ds-idrepo-1 12m 3630Mi ds-idrepo-2 13m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1177Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-799cdc5bdb-7t5x7 1m 54Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1m 6986Mi 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "status": "success", 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 DEBUG "status": "success", 05:33:45 INFO Thread started 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG { 05:33:45 DEBUG "data": { 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "data": { 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG { 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG --- stderr --- 05:33:45 DEBUG { 05:33:45 DEBUG "data": { 05:33:45 DEBUG "data": { 05:33:45 DEBUG { 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 DEBUG "data": { 05:33:45 INFO Thread started 05:33:45 INFO [loop_until]: OK (rc = 0) 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG { 05:33:45 DEBUG "data": { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG { 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG "resultType": "vector", 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "resultType": "vector", 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713587625" 05:33:45 DEBUG --- stdout --- 05:33:45 INFO Thread started 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "data": { 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "data": { 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "status": "success", 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG ----- output ----- 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG "data": { 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG "result": [ 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "data": { 05:33:45 DEBUG "result": [ 05:33:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6370Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 7389Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 11285Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3404Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 13693Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5212Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 11536Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 83m 0% 2933Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 68m 0% 10786Mi 9% gke-xlou-cdm-ds-0f601751-j488 71m 0% 6836Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 78m 0% 15645Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 102m 0% 7501Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13180Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 10834Mi 18% 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587625" 05:33:45 DEBUG "data": { 05:33:45 INFO Thread started 05:33:45 DEBUG "data": { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG { 05:33:45 DEBUG "data": { 05:33:45 DEBUG "result": [ 05:33:45 DEBUG { 05:33:45 DEBUG "data": { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "data": { 05:33:45 DEBUG "data": { 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG { 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG { 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG { 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG { 05:33:45 DEBUG --- stderr --- 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "resultType": "vector", 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587625" 05:33:45 INFO Thread started 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG --- http response --- 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG "metric": { 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "metric": { 05:33:45 DEBUG 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "result": [ 05:33:45 DEBUG { 05:33:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587625" 05:33:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:33:45 INFO Thread started 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:45 DEBUG "result": [ 05:33:45 DEBUG { 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "result": [ 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG "data": { 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG { 05:33:45 DEBUG "data": { 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:45 ERROR [http_cmd]: ERROR 05:33:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:45 DEBUG { 05:33:45 DEBUG "instance": "10.142.0.107:9100" 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG "metric": { 05:33:45 DEBUG }, 05:33:45 INFO All threads has been started 127.0.0.1 - - [20/Apr/2024 05:33:45] "GET /monitoring/start HTTP/1.1" 200 - 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "metric": { 05:33:45 DEBUG }, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "metric": { 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "metric": { 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG ----- output ----- 05:33:45 ERROR --- status code --- 05:33:45 DEBUG }, 05:33:45 DEBUG }, 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "metric": { 05:33:45 DEBUG }, 05:33:45 DEBUG "value": [ 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:33:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:45 DEBUG { 05:33:45 DEBUG 05:33:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "metric": { 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "pod": "ds-idrepo-0" 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "result": [ 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG { 05:33:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "value": [ 05:33:45 DEBUG { 05:33:45 DEBUG "instance": "10.142.0.107:9100" 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG "metric": { 05:33:45 DEBUG 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG "pod": "ds-cts-0" 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG }, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG "pod": "ds-idrepo-0" 05:33:45 DEBUG "instance": "10.142.0.107:9100" 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG }, 05:33:45 DEBUG { 05:33:45 DEBUG { 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:33:45 ERROR --- http response --- 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG }, 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG "pod": "ds-idrepo-0" 05:33:45 DEBUG 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG }, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG "metric": { 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 INFO [http_cmd]: http status code OK 05:33:45 DEBUG }, 05:33:45 DEBUG }, 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "data": { 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG { 05:33:45 DEBUG 05:33:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:33:45 DEBUG "data": { 05:33:45 DEBUG "value": [ 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG --- status code --- 05:33:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "data": { 05:33:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:45 DEBUG 05:33:45 DEBUG "data": { 05:33:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG 05:33:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG "value": [ 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG --- http response --- 05:33:45 DEBUG 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG { 05:33:45 DEBUG "data": { 05:33:45 DEBUG [...] 05:33:45 DEBUG "result": [ 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:33:45 DEBUG ----- output ----- 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG 05:33:45 DEBUG { 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG { 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "result": [ 05:33:45 DEBUG { 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG { 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG "pod": "ds-cts-0" 05:33:45 DEBUG "result": [ 05:33:45 DEBUG 05:33:45 DEBUG "metric": { 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:45 DEBUG "pod": "ds-cts-0" 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG { 05:33:45 DEBUG "status": "success", 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "metric": { 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG { 05:33:45 DEBUG 05:33:45 DEBUG "pod": "ds-cts-0" 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG "instance": "10.142.0.43:9100" 05:33:45 DEBUG }, 05:33:45 DEBUG "data": { 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "data": { 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG 05:33:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "pod": "ds-cts-0" 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "resultType": "vector", 05:33:45 DEBUG }, 05:33:45 DEBUG "pod": "ds-cts-0" 05:33:45 DEBUG }, 05:33:45 DEBUG }, 05:33:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "result": [ 05:33:45 DEBUG [...] 05:33:45 DEBUG }, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG "pod": "ds-idrepo-0" 05:33:45 DEBUG }, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "result": [ 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG }, 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:45 DEBUG 05:33:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:33:45 DEBUG "pod": "ds-idrepo-2" 05:33:45 DEBUG 05:33:45 DEBUG { 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG "instance": "10.142.0.43:9100" 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "0.004666666666670002" 05:33:45 DEBUG { 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "34470.59625212947" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "377387576.6950596" 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG }, 05:33:45 DEBUG "pod": "ds-idrepo-2" 05:33:45 DEBUG "metric": { 05:33:45 DEBUG 05:33:45 DEBUG "pod": "ds-cts-0" 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG "pod": "ds-idrepo-2" 05:33:45 DEBUG "pod": "ds-cts-2" 05:33:45 DEBUG ] 05:33:45 DEBUG "metric": { 05:33:45 DEBUG "0.021203254766196635" 05:33:45 DEBUG 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG ] 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG ] 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "instance": "10.142.0.43:9100" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "value": [ 05:33:45 DEBUG }, 05:33:45 DEBUG "instance": "10.142.0.107:9100" 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG "33214.58262350937" 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG }, 05:33:45 DEBUG }, 05:33:45 DEBUG } 05:33:45 DEBUG "instance": "10.142.0.107:9100" 05:33:45 DEBUG }, 05:33:45 DEBUG [...] 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG "7422210048" 05:33:45 DEBUG } 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG }, 05:33:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG [...] 05:33:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG ] 05:33:45 DEBUG [...] 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "value": [ 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG "0" 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG "0" 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG "0" 05:33:45 DEBUG 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "27275.666666666664" 05:33:45 DEBUG "0" 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG "0.013024974443824418" 05:33:45 DEBUG 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG ] 05:33:45 DEBUG 05:33:45 DEBUG "pod": "ds-cts-2" 05:33:45 DEBUG }, 05:33:45 DEBUG "0" 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG [...] 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG ] 05:33:45 DEBUG 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "pod": "ds-cts-2" 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG "34308.53333333333" 05:33:45 DEBUG "0" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG "value": [ 05:33:45 DEBUG ] 05:33:45 DEBUG [...] 05:33:45 DEBUG } 05:33:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:45 DEBUG "0" 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "0" 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG }, 05:33:45 DEBUG 05:33:45 DEBUG ] 05:33:45 DEBUG 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG ] 05:33:45 DEBUG }, 05:33:45 DEBUG ] 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG "value": [ 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG "0" 05:33:45 DEBUG } 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG "value": [ 05:33:45 DEBUG } 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "pod": "ds-cts-2" 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG "instance": "10.142.0.43:9100" 05:33:45 DEBUG } 05:33:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG 05:33:45 DEBUG } 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG ] 05:33:45 DEBUG "0" 05:33:45 DEBUG }, 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG }, 05:33:45 DEBUG }, 05:33:45 DEBUG } 05:33:45 DEBUG "instance": "10.142.0.43:9100" 05:33:45 DEBUG "0.091" 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG "value": [ 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG "0" 05:33:45 DEBUG "value": [ 05:33:45 DEBUG "value": [ 05:33:45 DEBUG ] 05:33:45 DEBUG }, 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG } 05:33:45 DEBUG "value": [ 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG "0" 05:33:45 DEBUG } 05:33:45 DEBUG "3036923038720" 05:33:45 DEBUG "1" 05:33:45 DEBUG } 05:33:45 DEBUG 1713587625, 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG "0.007033333333316705" 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG ] 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:33:45 DEBUG } 05:34:45 INFO 05:34:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:34:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:45 INFO 05:34:45 INFO [loop_until]: kubectl --namespace=xlou top node 05:34:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587685" 05:34:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587685" 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 ERROR [http_cmd]: ERROR 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [loop_until]: OK (rc = 0) 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 DEBUG --- status code --- 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [loop_until]: OK (rc = 0) 05:34:45 ERROR --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 DEBUG --- status code --- 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 DEBUG --- stdout --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- stdout --- 05:34:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1763m 8600Mi am-767f6b67b5-qcqz2 326m 8675Mi am-767f6b67b5-vbltz 343m 8602Mi ds-cts-0 283m 10566Mi ds-cts-1 117m 10674Mi ds-cts-2 524m 10603Mi ds-idrepo-0 92m 3752Mi ds-idrepo-1 919m 3631Mi ds-idrepo-2 484m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 2384m 7303Mi 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 ERROR --- http response --- 05:34:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 6223Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 925m 3% 7419Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 602m 2% 11293Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3397Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 565m 1% 13672Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 137m 0% 5223Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 629m 2% 11550Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 1030m 3% 2951Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2207m 7% 10834Mi 9% gke-xlou-cdm-ds-0f601751-j488 360m 1% 6858Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 325m 1% 15607Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 1005m 3% 7519Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 387m 1% 13178Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 2928m 18% 11139Mi 18% 05:34:45 DEBUG --- http response --- 05:34:45 INFO [http_cmd]: http status code OK 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG --- stderr --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:34:45 DEBUG --- stderr --- 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:34:45 DEBUG --- status code --- 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:34:45 DEBUG { 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG { 05:34:45 DEBUG --- http response --- 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG { 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG { 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG ----- output ----- 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "data": { 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG { 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "data": { 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "data": { 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "status": "success", 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG "data": { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "result": [ 05:34:45 DEBUG { 05:34:45 DEBUG "result": [ 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "resultType": "vector", 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "pod": "ds-idrepo-0" 05:34:45 DEBUG { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:45 DEBUG "instance": "10.142.0.107:9100" 05:34:45 DEBUG "result": [ 05:34:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "metric": { 05:34:45 DEBUG }, 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG "instance": "10.142.0.107:9100" 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:34:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:34:45 DEBUG "pod": "ds-idrepo-0" 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG "pod": "ds-cts-0" 05:34:45 DEBUG "instance": "10.142.0.107:9100" 05:34:45 DEBUG "pod": "ds-cts-0" 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG "pod": "ds-cts-0" 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG { 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG "pod": "ds-idrepo-0" 05:34:45 DEBUG "instance": "10.142.0.107:9100" 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG "pod": "ds-idrepo-0" 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "metric": { 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "instance": "10.142.0.107:9100" 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG 05:34:45 DEBUG "value": [ 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG "value": [ 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "instance": "10.142.0.43:9100" 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "pod": "ds-idrepo-2" 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "pod": "ds-cts-2" 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:45 DEBUG 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:34:45 DEBUG }, 05:34:45 DEBUG [...] 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG 05:34:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:45 DEBUG 05:34:45 DEBUG 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:34:45 DEBUG "pod": "ds-idrepo-2" 05:34:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG "pod": "ds-cts-2" 05:34:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:45 DEBUG "pod": "ds-cts-2" 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG "instance": "10.142.0.43:9100" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:34:45 DEBUG "value": [ 05:34:45 DEBUG 05:34:45 DEBUG "pod": "ds-cts-2" 05:34:45 DEBUG "instance": "10.142.0.43:9100" 05:34:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:45 DEBUG "pod": "ds-idrepo-2" 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:34:45 DEBUG }, 05:34:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG "instance": "10.142.0.43:9100" 05:34:45 DEBUG "value": [ 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:34:45 DEBUG "value": [ 05:34:45 DEBUG }, 05:34:45 DEBUG "0" 05:34:45 DEBUG }, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG }, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG }, 05:34:45 DEBUG }, 05:34:45 DEBUG "31683189.666666664" 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG }, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG }, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG ] 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "224811.81431522086" 05:34:45 DEBUG 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "0" 05:34:45 DEBUG "value": [ 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "value": [ 05:34:45 DEBUG ] 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "0" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "value": [ 05:34:45 DEBUG "228.9" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG } 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "6125.6625154011135" 05:34:45 DEBUG "instance": "10.142.0.43:9100" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG ] 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG "0.091" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG } 05:34:45 DEBUG "7776624640" 05:34:45 DEBUG "0" 05:34:45 DEBUG "228.93333333333334" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG ] 05:34:45 DEBUG "0" 05:34:45 DEBUG "0" 05:34:45 DEBUG "0" 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG ] 05:34:45 DEBUG "0.02799999999999727" 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG }, 05:34:45 DEBUG "0" 05:34:45 DEBUG "3037259512832" 05:34:45 DEBUG "3422.5" 05:34:45 DEBUG "0" 05:34:45 DEBUG } 05:34:45 DEBUG "0" 05:34:45 DEBUG ] 05:34:45 DEBUG "228.5" 05:34:45 DEBUG "0.6716606776273643" 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG "31687280.866666667" 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG "0" 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG "value": [ 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG 1713587685, 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG "0.03846666666676886" 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG ] 05:34:45 DEBUG } 05:34:45 DEBUG } 05:34:45 DEBUG } 05:35:45 INFO 05:35:45 INFO [loop_until]: kubectl --namespace=xlou top node 05:35:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO 05:35:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713587745" 05:35:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:35:45 ERROR [http_cmd]: ERROR 05:35:45 ERROR --- status code --- 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:35:45 ERROR --- http response --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [loop_until]: OK (rc = 0) 05:35:45 DEBUG --- stdout --- 05:35:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6196Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7012m 23% 7424Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14287m 47% 11324Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3403Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8086m 27% 13783Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 14454m 48% 11557Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 17789m 59% 2998Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 15902m 53% 10844Mi 9% gke-xlou-cdm-ds-0f601751-j488 7271m 24% 6865Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 7384m 24% 15737Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7061m 23% 7531Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8312m 27% 13279Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8202m 51% 11185Mi 19% 05:35:45 DEBUG --- stderr --- 05:35:45 DEBUG 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "0" 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "0" 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [loop_until]: OK (rc = 0) 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 DEBUG --- stdout --- 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 14170m 8617Mi am-767f6b67b5-qcqz2 14482m 8680Mi am-767f6b67b5-vbltz 15387m 8624Mi ds-cts-0 8126m 10620Mi ds-cts-1 7828m 10719Mi ds-cts-2 8102m 10660Mi ds-idrepo-0 6799m 3752Mi ds-idrepo-1 6691m 3632Mi ds-idrepo-2 7083m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7903m 7351Mi 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 DEBUG --- stderr --- 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587745" 05:35:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713587745" 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- status code --- 05:35:45 INFO [http_cmd]: http status code OK 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG --- status code --- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:45 DEBUG { 05:35:45 DEBUG --- http response --- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:35:45 DEBUG "data": { 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "data": { 05:35:45 DEBUG { 05:35:45 DEBUG "data": { 05:35:45 DEBUG ----- output ----- 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG { 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "data": { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "status": "success", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "data": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG { 05:35:45 DEBUG "data": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "pod": "ds-cts-0" 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "metric": { 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "resultType": "vector", 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "result": [ 05:35:45 DEBUG }, 05:35:45 DEBUG "result": [ 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:45 DEBUG { 05:35:45 DEBUG { 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG }, 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG }, 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:45 DEBUG { 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG "result": [ 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG { 05:35:45 DEBUG "value": [ 05:35:45 DEBUG { 05:35:45 DEBUG "pod": "ds-idrepo-0" 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG "pod": "ds-cts-0" 05:35:45 DEBUG }, 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "instance": "10.142.0.107:9100" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "instance": "10.142.0.107:9100" 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "instance": "10.142.0.107:9100" 05:35:45 DEBUG }, 05:35:45 DEBUG "metric": { 05:35:45 DEBUG }, 05:35:45 DEBUG { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG "metric": { 05:35:45 DEBUG 05:35:45 DEBUG "metric": { 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "instance": "10.142.0.107:9100" 05:35:45 DEBUG "pod": "ds-idrepo-0" 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "pod": "ds-cts-0" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:35:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:35:45 DEBUG "pod": "ds-idrepo-0" 05:35:45 DEBUG "pod": "ds-idrepo-0" 05:35:45 DEBUG 05:35:45 DEBUG "instance": "10.142.0.107:9100" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG [...] 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG [...] 05:35:45 DEBUG "pod": "ds-cts-2" 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG [...] 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG "pod": "ds-cts-2" 05:35:45 DEBUG 05:35:45 DEBUG "instance": "10.142.0.43:9100" 05:35:45 DEBUG 05:35:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:35:45 DEBUG 05:35:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG "pod": "ds-idrepo-2" 05:35:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG }, 05:35:45 DEBUG "instance": "10.142.0.43:9100" 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG "instance": "10.142.0.43:9100" 05:35:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:35:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "instance": "10.142.0.43:9100" 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 05:35:45 DEBUG "value": [ 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG }, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "pod": "ds-cts-2" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:35:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:35:45 DEBUG "pod": "ds-idrepo-2" 05:35:45 DEBUG "pod": "ds-cts-2" 05:35:45 DEBUG "0" 05:35:45 DEBUG "instance": "10.142.0.43:9100" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "pod": "ds-idrepo-2" 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "2662.9" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG }, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG }, 05:35:45 DEBUG ] 05:35:45 DEBUG }, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "0.0003105222222222222" 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "63955282.233333334" 05:35:45 DEBUG }, 05:35:45 DEBUG "0.091" 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG ] 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG "0" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "2662.8333333333335" 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "value": [ 05:35:45 DEBUG } 05:35:45 DEBUG "value": [ 05:35:45 DEBUG "0" 05:35:45 DEBUG "0" 05:35:45 DEBUG "2662.9333333333334" 05:35:45 DEBUG "561.9233803203348" 05:35:45 DEBUG ] 05:35:45 DEBUG "0.04533333333332621" 05:35:45 DEBUG ] 05:35:45 DEBUG "value": [ 05:35:45 DEBUG ] 05:35:45 DEBUG "3039813864448" 05:35:45 DEBUG "112913.5" 05:35:45 DEBUG "0" 05:35:45 DEBUG } 05:35:45 DEBUG "63959840.46666667" 05:35:45 DEBUG ] 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG ] 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG ] 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG 1713587745, 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG "0" 05:35:45 DEBUG } 05:35:45 DEBUG "7804338176" 05:35:45 DEBUG "7.321270736912232" 05:35:45 DEBUG "0" 05:35:45 DEBUG "0" 05:35:45 DEBUG } 05:35:45 DEBUG "0.06329999999996688" 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG "0" 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG ] 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:35:45 DEBUG } 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO 05:36:45 INFO [loop_until]: kubectl --namespace=xlou top node 05:36:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:45 ERROR [http_cmd]: ERROR 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 ERROR --- status code --- 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 DEBUG --- status code --- 05:36:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 ERROR --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:36:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:36:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "118465733.97811174" 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "0" 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 INFO [loop_until]: OK (rc = 0) 05:36:45 DEBUG --- stdout --- 05:36:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6196Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6972m 23% 7387Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14238m 47% 11311Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3385Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8176m 27% 13802Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5212Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 14565m 48% 11556Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18387m 61% 2993Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 15809m 52% 10830Mi 9% gke-xlou-cdm-ds-0f601751-j488 7242m 24% 6860Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 7721m 25% 15743Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 6969m 23% 7527Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8112m 27% 13286Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8181m 51% 11185Mi 19% 05:36:45 DEBUG --- stderr --- 05:36:45 DEBUG 05:36:45 INFO 05:36:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:36:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "4.217164148427195" 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "instance": "10.142.0.107:9100" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "instance": "10.142.0.43:9100" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "63254694.16666667" 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "instance": "10.142.0.107:9100" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "instance": "10.142.0.43:9100" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "0.04066666666668464" 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 INFO [loop_until]: OK (rc = 0) 05:36:45 DEBUG --- stdout --- 05:36:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 13887m 8617Mi am-767f6b67b5-qcqz2 14654m 8681Mi am-767f6b67b5-vbltz 15265m 8625Mi ds-cts-0 8350m 10642Mi ds-cts-1 7755m 10731Mi ds-cts-2 8361m 10671Mi ds-idrepo-0 6910m 3719Mi ds-idrepo-1 6617m 3632Mi ds-idrepo-2 6955m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 12m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7988m 7352Mi 05:36:45 DEBUG --- stderr --- 05:36:45 DEBUG 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587805" 05:36:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587805" 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:45 INFO [http_cmd]: http status code OK 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG --- status code --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:36:45 DEBUG { 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG --- http response --- 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:36:45 DEBUG { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG ----- output ----- 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG "data": { 05:36:45 DEBUG "data": { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "status": "success", 05:36:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:36:45 DEBUG "result": [ 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "pod": "ds-cts-0" 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "result": [ 05:36:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:45 DEBUG "instance": "10.142.0.107:9100" 05:36:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:45 DEBUG "instance": "10.142.0.107:9100" 05:36:45 DEBUG "pod": "ds-cts-0" 05:36:45 DEBUG "data": { 05:36:45 DEBUG }, 05:36:45 DEBUG { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:36:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:45 DEBUG "pod": "ds-cts-0" 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG { 05:36:45 DEBUG { 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "resultType": "vector", 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "metric": { 05:36:45 DEBUG "metric": { 05:36:45 DEBUG }, 05:36:45 DEBUG 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG }, 05:36:45 DEBUG 05:36:45 DEBUG "result": [ 05:36:45 DEBUG 05:36:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG { 05:36:45 DEBUG 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "metric": { 05:36:45 DEBUG 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG "instance": "10.142.0.107:9100" 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG }, 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG [...] 05:36:45 DEBUG [...] 05:36:45 DEBUG 05:36:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "instance": "10.142.0.43:9100" 05:36:45 DEBUG 05:36:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "pod": "ds-idrepo-2" 05:36:45 DEBUG "pod": "ds-cts-2" 05:36:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG }, 05:36:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG 05:36:45 DEBUG }, 05:36:45 DEBUG 05:36:45 DEBUG }, 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "pod": "ds-idrepo-0" 05:36:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:45 DEBUG "pod": "ds-cts-2" 05:36:45 DEBUG "pod": "ds-idrepo-2" 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG "instance": "10.142.0.43:9100" 05:36:45 DEBUG "value": [ 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG "pod": "ds-cts-2" 05:36:45 DEBUG "value": [ 05:36:45 DEBUG [...] 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG "pod": "ds-cts-2" 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 05:36:45 DEBUG 05:36:45 DEBUG }, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG }, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "pod": "ds-idrepo-2" 05:36:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "272943" 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "3042789753856" 05:36:45 DEBUG 05:36:45 DEBUG "8.029307682075626" 05:36:45 DEBUG }, 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "2700.32936138675" 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "0" 05:36:45 DEBUG "0" 05:36:45 DEBUG "2659.1666666666665" 05:36:45 DEBUG }, 05:36:45 DEBUG }, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "63250240.86666667" 05:36:45 DEBUG "2659.3333333333335" 05:36:45 DEBUG "0" 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG ] 05:36:45 DEBUG 05:36:45 DEBUG ] 05:36:45 DEBUG "value": [ 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG ] 05:36:45 DEBUG "0" 05:36:45 DEBUG "0" 05:36:45 DEBUG "0" 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG "value": [ 05:36:45 DEBUG "value": [ 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG "0.00016666666666666663" 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG "0" 05:36:45 DEBUG "0.091" 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG 1713587805, 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG "instance": "10.142.0.43:9100" 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG "7806160896" 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG "0" 05:36:45 DEBUG "2659.5333333333333" 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG }, 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG "value": [ 05:36:45 DEBUG } 05:36:45 DEBUG } 05:36:45 DEBUG ] 05:36:45 DEBUG ] 05:36:45 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG 1713587805, 05:36:46 DEBUG } 05:36:46 DEBUG ] 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG ] 05:36:46 DEBUG ] 05:36:46 DEBUG } 05:36:46 DEBUG "0.0567666666666156" 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG ] 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG } 05:36:46 DEBUG ] 05:36:46 DEBUG } 05:36:46 DEBUG } 05:37:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587865" 05:37:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587865" 05:37:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713587865" 05:37:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587865" 05:37:45 ERROR [http_cmd]: ERROR 05:37:45 INFO [http_cmd]: http status code OK 05:37:45 DEBUG --- status code --- 05:37:45 ERROR --- status code --- 05:37:45 INFO [http_cmd]: http status code OK 05:37:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:37:45 DEBUG --- status code --- 05:37:45 DEBUG --- http response --- 05:37:45 ERROR --- http response --- 05:37:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:37:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:45 DEBUG --- http response --- 05:37:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:37:45 DEBUG ----- output ----- 05:37:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:37:45 DEBUG { 05:37:45 DEBUG "status": "success", 05:37:45 DEBUG "data": { 05:37:45 DEBUG "resultType": "vector", 05:37:45 DEBUG "result": [ 05:37:45 DEBUG { 05:37:45 DEBUG "metric": { 05:37:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG [...] 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 1713587865, 05:37:45 DEBUG "228.33333333333334" 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG } 05:37:45 INFO [http_cmd]: http status code OK 05:37:45 DEBUG --- status code --- 05:37:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:45 DEBUG --- http response --- 05:37:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:37:45 DEBUG ----- output ----- 05:37:45 DEBUG { 05:37:45 DEBUG "status": "success", 05:37:45 DEBUG "data": { 05:37:45 DEBUG "resultType": "vector", 05:37:45 DEBUG "result": [ 05:37:45 DEBUG { 05:37:45 DEBUG "metric": { 05:37:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG [...] 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG "pod": "ds-cts-2" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 1713587865, 05:37:45 DEBUG "68420.66283371765" 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG } 05:37:45 INFO 05:37:45 INFO [loop_until]: kubectl --namespace=xlou top node 05:37:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587865" 05:37:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587865" 05:37:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587865" 05:37:45 INFO 05:37:45 INFO [loop_until]: kubectl --namespace=xlou top pods 05:37:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:37:45 INFO [http_cmd]: http status code OK 05:37:45 DEBUG --- status code --- 05:37:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:45 DEBUG --- http response --- 05:37:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:37:45 DEBUG ----- output ----- 05:37:45 DEBUG { 05:37:45 DEBUG "status": "success", 05:37:45 DEBUG "data": { 05:37:45 DEBUG "resultType": "vector", 05:37:45 DEBUG "result": [ 05:37:45 DEBUG { 05:37:45 DEBUG "metric": { 05:37:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG [...] 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG "pod": "ds-cts-2" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 1713587865, 05:37:45 DEBUG "4.253587156279933" 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG } 05:37:45 INFO [http_cmd]: http status code OK 05:37:45 DEBUG --- status code --- 05:37:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:45 DEBUG --- http response --- 05:37:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:45 DEBUG ----- output ----- 05:37:45 DEBUG { 05:37:45 DEBUG "status": "success", 05:37:45 DEBUG "data": { 05:37:45 DEBUG "resultType": "vector", 05:37:45 DEBUG "result": [ 05:37:45 DEBUG { 05:37:45 DEBUG "metric": { 05:37:45 DEBUG "instance": "10.142.0.107:9100" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG [...] 05:37:45 DEBUG 05:37:45 INFO [http_cmd]: http status code OK 05:37:45 DEBUG 05:37:45 DEBUG --- status code --- 05:37:45 DEBUG 05:37:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:45 DEBUG "instance": "10.142.0.43:9100" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 1713587865, 05:37:45 DEBUG "72045250.8" 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG } 05:37:45 DEBUG --- http response --- 05:37:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:45 DEBUG ----- output ----- 05:37:45 DEBUG { 05:37:45 DEBUG "status": "success", 05:37:45 DEBUG "data": { 05:37:45 DEBUG "resultType": "vector", 05:37:45 DEBUG "result": [ 05:37:45 DEBUG { 05:37:45 DEBUG "metric": { 05:37:45 DEBUG "instance": "10.142.0.107:9100" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG [...] 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG 05:37:45 DEBUG "instance": "10.142.0.43:9100" 05:37:45 DEBUG }, 05:37:45 DEBUG "value": [ 05:37:45 DEBUG 1713587865, 05:37:45 DEBUG "0.036999999999998555" 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG ] 05:37:45 DEBUG } 05:37:45 DEBUG } 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "pod": "ds-idrepo-0" 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "pod": "ds-idrepo-2" 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG "0" 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713587866" 05:37:46 INFO [loop_until]: OK (rc = 0) 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [loop_until]: OK (rc = 0) 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 DEBUG --- stdout --- 05:37:46 DEBUG --- status code --- 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- stdout --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 15011m 8622Mi am-767f6b67b5-qcqz2 14854m 8685Mi am-767f6b67b5-vbltz 15033m 8626Mi ds-cts-0 9140m 10646Mi ds-cts-1 8110m 10735Mi ds-cts-2 9100m 10676Mi ds-idrepo-0 6947m 3720Mi ds-idrepo-1 7078m 3618Mi ds-idrepo-2 7223m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 12m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7972m 7357Mi 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG --- status code --- 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 INFO [http_cmd]: http status code OK 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6196Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7115m 23% 7382Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14554m 48% 11319Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3369Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 9298m 31% 13791Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 5223Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 14214m 47% 11558Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18774m 62% 2985Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 16063m 53% 10828Mi 9% gke-xlou-cdm-ds-0f601751-j488 7323m 24% 6853Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 8273m 27% 15744Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7233m 24% 7497Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9099m 30% 13286Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8184m 51% 11185Mi 19% 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- stderr --- 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG --- status code --- 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- stderr --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:46 DEBUG 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:46 DEBUG 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG --- http response --- 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG { 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG { 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG { 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:37:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG ----- output ----- 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "data": { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "data": { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "data": { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "status": "success", 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "result": [ 05:37:46 DEBUG { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "data": { 05:37:46 DEBUG "data": { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "result": [ 05:37:46 DEBUG { 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "result": [ 05:37:46 DEBUG { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "result": [ 05:37:46 DEBUG { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "result": [ 05:37:46 DEBUG { 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "result": [ 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "pod": "ds-cts-0" 05:37:46 DEBUG }, 05:37:46 DEBUG { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG { 05:37:46 DEBUG { 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "instance": "10.142.0.107:9100" 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "pod": "ds-idrepo-0" 05:37:46 DEBUG { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG "resultType": "vector", 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "metric": { 05:37:46 DEBUG { 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG "instance": "10.142.0.107:9100" 05:37:46 DEBUG }, 05:37:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:37:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG }, 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "pod": "ds-cts-0" 05:37:46 DEBUG "pod": "ds-idrepo-0" 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG }, 05:37:46 DEBUG 05:37:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:37:46 DEBUG }, 05:37:46 DEBUG "result": [ 05:37:46 DEBUG "pod": "ds-idrepo-0" 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG "metric": { 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "value": [ 05:37:46 DEBUG }, 05:37:46 DEBUG "instance": "10.142.0.107:9100" 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG { 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG "pod": "ds-cts-0" 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG "value": [ 05:37:46 DEBUG }, 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "value": [ 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG "metric": { 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "value": [ 05:37:46 DEBUG }, 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "pod": "ds-cts-2" 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "pod": "ds-idrepo-2" 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "pod": "ds-idrepo-2" 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "instance": "10.142.0.43:9100" 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:37:46 DEBUG }, 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:46 DEBUG [...] 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:46 DEBUG "instance": "10.142.0.43:9100" 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG "0" 05:37:46 DEBUG 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:37:46 DEBUG }, 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:37:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG "0" 05:37:46 DEBUG }, 05:37:46 DEBUG ] 05:37:46 DEBUG 05:37:46 DEBUG "0" 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG 05:37:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:46 DEBUG "pod": "ds-cts-2" 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "72041092.13333333" 05:37:46 DEBUG }, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG ] 05:37:46 DEBUG "pod": "ds-cts-2" 05:37:46 DEBUG "instance": "10.142.0.43:9100" 05:37:46 DEBUG "value": [ 05:37:46 DEBUG } 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG ] 05:37:46 DEBUG 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG ] 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "0" 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG } 05:37:46 DEBUG }, 05:37:46 DEBUG }, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG ] 05:37:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:37:46 DEBUG }, 05:37:46 DEBUG "value": [ 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG } 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "2676.1" 05:37:46 DEBUG "0.091" 05:37:46 DEBUG "3045798364160" 05:37:46 DEBUG } 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "1011.7015035969001" 05:37:46 DEBUG } 05:37:46 DEBUG "2675.8" 05:37:46 DEBUG "value": [ 05:37:46 DEBUG "pod": "ds-cts-2" 05:37:46 DEBUG } 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG "0" 05:37:46 DEBUG ] 05:37:46 DEBUG "8.02298974841389" 05:37:46 DEBUG } 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG }, 05:37:46 DEBUG "429583" 05:37:46 DEBUG "2675.4333333333334" 05:37:46 DEBUG "0" 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG "0" 05:37:46 DEBUG "0.060166666666701944" 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG "7811452928" 05:37:46 DEBUG "value": [ 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG 1713587866, 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG "0.0002" 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG } 05:37:46 DEBUG ] 05:37:46 DEBUG } 05:37:46 DEBUG } 05:38:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713587925" 05:38:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587925" 05:38:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587925" 05:38:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587925" 05:38:45 INFO [http_cmd]: http status code OK 05:38:45 ERROR [http_cmd]: ERROR 05:38:45 INFO [http_cmd]: http status code OK 05:38:45 DEBUG --- status code --- 05:38:45 ERROR --- status code --- 05:38:45 DEBUG --- status code --- 05:38:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:38:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:45 DEBUG --- http response --- 05:38:45 ERROR --- http response --- 05:38:45 DEBUG --- http response --- 05:38:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:38:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:38:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:38:45 DEBUG ----- output ----- 05:38:45 DEBUG { 05:38:45 DEBUG "status": "success", 05:38:45 DEBUG "data": { 05:38:45 DEBUG "resultType": "vector", 05:38:45 DEBUG "result": [ 05:38:45 DEBUG { 05:38:45 DEBUG "metric": { 05:38:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:45 DEBUG }, 05:38:45 DEBUG "value": [ 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG [...] 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:45 DEBUG }, 05:38:45 DEBUG "value": [ 05:38:45 DEBUG 1713587925, 05:38:45 DEBUG "2662.9333333333334" 05:38:45 DEBUG ] 05:38:45 DEBUG } 05:38:45 DEBUG ] 05:38:45 DEBUG } 05:38:45 DEBUG } 05:38:45 INFO [http_cmd]: http status code OK 05:38:45 DEBUG --- status code --- 05:38:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:45 DEBUG --- http response --- 05:38:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:38:45 DEBUG ----- output ----- 05:38:45 DEBUG { 05:38:45 DEBUG "status": "success", 05:38:45 DEBUG "data": { 05:38:45 DEBUG "resultType": "vector", 05:38:45 DEBUG "result": [ 05:38:45 DEBUG { 05:38:45 DEBUG "metric": { 05:38:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:45 DEBUG }, 05:38:45 DEBUG "value": [ 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG [...] 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG 05:38:45 DEBUG "pod": "ds-cts-2" 05:38:45 DEBUG }, 05:38:45 DEBUG "value": [ 05:38:45 DEBUG 1713587925, 05:38:45 DEBUG "0" 05:38:45 DEBUG ] 05:38:45 DEBUG } 05:38:45 DEBUG ] 05:38:45 DEBUG } 05:38:45 DEBUG } 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587926" 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-cts-2" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "0" 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG ] 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG } 05:38:46 DEBUG { 05:38:46 DEBUG } 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "instance": "10.142.0.107:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "instance": "10.142.0.43:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "80802032.36666667" 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "instance": "10.142.0.107:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "instance": "10.142.0.43:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "0.07300000000001317" 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "pod": "ds-idrepo-0" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-idrepo-2" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "0" 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 INFO 05:38:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:38:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:46 INFO 05:38:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:38:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "0" 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 DEBUG --- status code --- 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713587926" 05:38:46 DEBUG --- status code --- 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [loop_until]: OK (rc = 0) 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587926" 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587926" 05:38:46 DEBUG --- stdout --- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [loop_until]: OK (rc = 0) 05:38:46 DEBUG --- status code --- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- http response --- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- http response --- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17720m 8639Mi am-767f6b67b5-qcqz2 16876m 8704Mi am-767f6b67b5-vbltz 18809m 8645Mi ds-cts-0 17750m 10630Mi ds-cts-1 16327m 10736Mi ds-cts-2 17947m 10652Mi ds-idrepo-0 7957m 3720Mi ds-idrepo-1 7755m 3611Mi ds-idrepo-2 7948m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7903m 7359Mi 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG --- status code --- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG --- stdout --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG --- status code --- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG --- stderr --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6206Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7947m 26% 7386Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17348m 58% 11345Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3376Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17778m 59% 13757Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 5215Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17716m 59% 11596Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18896m 63% 3000Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19370m 64% 10862Mi 9% gke-xlou-cdm-ds-0f601751-j488 8161m 27% 6847Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16563m 55% 15704Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7905m 26% 7505Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18407m 61% 13260Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8175m 51% 11187Mi 19% 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG ----- output ----- 05:38:46 INFO [http_cmd]: http status code OK 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG --- stderr --- 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG { 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG "data": { 05:38:46 DEBUG --- status code --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG "data": { 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG "result": [ 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG --- http response --- 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG { 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:38:46 DEBUG { 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG { 05:38:46 DEBUG { 05:38:46 DEBUG { 05:38:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG ----- output ----- 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "data": { 05:38:46 DEBUG { 05:38:46 DEBUG { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "data": { 05:38:46 DEBUG { 05:38:46 DEBUG { 05:38:46 DEBUG "data": { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "data": { 05:38:46 DEBUG "pod": "ds-cts-0" 05:38:46 DEBUG "data": { 05:38:46 DEBUG { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG }, 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "status": "success", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "data": { 05:38:46 DEBUG "data": { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "pod": "ds-idrepo-0" 05:38:46 DEBUG "data": { 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "data": { 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG { 05:38:46 DEBUG }, 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG { 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG { 05:38:46 DEBUG 05:38:46 DEBUG { 05:38:46 DEBUG "resultType": "vector", 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "result": [ 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "pod": "ds-idrepo-0" 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "metric": { 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "result": [ 05:38:46 DEBUG "instance": "10.142.0.107:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:46 DEBUG { 05:38:46 DEBUG { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "pod": "ds-idrepo-0" 05:38:46 DEBUG }, 05:38:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:46 DEBUG 05:38:46 DEBUG { 05:38:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "instance": "10.142.0.107:9100" 05:38:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:46 DEBUG 05:38:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:38:46 DEBUG { 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG }, 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "instance": "10.142.0.107:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG }, 05:38:46 DEBUG 05:38:46 DEBUG "metric": { 05:38:46 DEBUG }, 05:38:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:46 DEBUG }, 05:38:46 DEBUG }, 05:38:46 DEBUG [...] 05:38:46 DEBUG }, 05:38:46 DEBUG "metric": { 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:38:46 DEBUG "pod": "ds-cts-0" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-cts-0" 05:38:46 DEBUG "value": [ 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG }, 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG }, 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG }, 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-cts-2" 05:38:46 DEBUG }, 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG [...] 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-idrepo-2" 05:38:46 DEBUG "pod": "ds-idrepo-2" 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG [...] 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG }, 05:38:46 DEBUG }, 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "0" 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:46 DEBUG [...] 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG ] 05:38:46 DEBUG 05:38:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:46 DEBUG 05:38:46 DEBUG "instance": "10.142.0.43:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-cts-2" 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "instance": "10.142.0.43:9100" 05:38:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:46 DEBUG } 05:38:46 DEBUG 05:38:46 DEBUG }, 05:38:46 DEBUG 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 05:38:46 DEBUG 05:38:46 DEBUG "instance": "10.142.0.43:9100" 05:38:46 DEBUG }, 05:38:46 DEBUG "0" 05:38:46 DEBUG "0" 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-cts-2" 05:38:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:38:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:38:46 DEBUG }, 05:38:46 DEBUG }, 05:38:46 DEBUG ] 05:38:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG 05:38:46 DEBUG "pod": "ds-cts-2" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:46 DEBUG }, 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG } 05:38:46 DEBUG }, 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "480560" 05:38:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "2672.0333333333333" 05:38:46 DEBUG "value": [ 05:38:46 DEBUG "0.092" 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG ] 05:38:46 DEBUG }, 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "3049178829824" 05:38:46 DEBUG "0" 05:38:46 DEBUG ] 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "7814410240" 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG ] 05:38:46 DEBUG "0" 05:38:46 DEBUG "80797103.96666667" 05:38:46 DEBUG } 05:38:46 DEBUG "value": [ 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "0.0006" 05:38:46 DEBUG ] 05:38:46 DEBUG "0.18213333333333706" 05:38:46 DEBUG "0" 05:38:46 DEBUG } 05:38:46 DEBUG "8.026763873111776" 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG 1713587926, 05:38:46 DEBUG "2671.4" 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG "2674.233333333333" 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG "0" 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG ] 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:38:46 DEBUG } 05:39:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587985" 05:39:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587985" 05:39:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713587985" 05:39:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587985" 05:39:45 INFO [http_cmd]: http status code OK 05:39:45 DEBUG --- status code --- 05:39:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:45 DEBUG --- http response --- 05:39:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:39:45 DEBUG ----- output ----- 05:39:45 DEBUG { 05:39:45 DEBUG "status": "success", 05:39:45 DEBUG "data": { 05:39:45 DEBUG "resultType": "vector", 05:39:45 DEBUG "result": [ 05:39:45 DEBUG { 05:39:45 DEBUG "metric": { 05:39:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:45 DEBUG }, 05:39:45 DEBUG "value": [ 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG [...] 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG "pod": "ds-cts-2" 05:39:45 DEBUG }, 05:39:45 DEBUG "value": [ 05:39:45 DEBUG 1713587985, 05:39:45 DEBUG "718857.7858083261" 05:39:45 DEBUG ] 05:39:45 DEBUG } 05:39:45 DEBUG ] 05:39:45 DEBUG } 05:39:45 DEBUG } 05:39:45 INFO [http_cmd]: http status code OK 05:39:45 INFO [http_cmd]: http status code OK 05:39:45 DEBUG --- status code --- 05:39:45 DEBUG --- status code --- 05:39:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:45 DEBUG --- http response --- 05:39:45 DEBUG --- http response --- 05:39:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:39:45 DEBUG ----- output ----- 05:39:45 DEBUG { 05:39:45 DEBUG "status": "success", 05:39:45 DEBUG "data": { 05:39:45 DEBUG "resultType": "vector", 05:39:45 DEBUG "result": [ 05:39:45 DEBUG { 05:39:45 DEBUG "metric": { 05:39:45 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:45 DEBUG }, 05:39:45 DEBUG "value": [ 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG [...] 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG 05:39:45 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:45 DEBUG }, 05:39:45 DEBUG "value": [ 05:39:45 DEBUG 1713587985, 05:39:45 DEBUG "2652.0666666666666" 05:39:45 DEBUG ] 05:39:45 DEBUG } 05:39:45 DEBUG ] 05:39:45 DEBUG } 05:39:45 DEBUG } 05:39:45 ERROR [http_cmd]: ERROR 05:39:45 ERROR --- status code --- 05:39:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:39:45 ERROR --- http response --- 05:39:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:39:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587986" 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "instance": "10.142.0.107:9100" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "instance": "10.142.0.43:9100" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "80468498.56666666" 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "pod": "ds-cts-2" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "10.761042384729672" 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG } 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "instance": "10.142.0.107:9100" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "instance": "10.142.0.43:9100" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0.06699999999999778" 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "pod": "ds-idrepo-0" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "pod": "ds-idrepo-2" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0" 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0" 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 INFO 05:39:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:39:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:46 INFO 05:39:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:39:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713587986" 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713587986" 05:39:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713587986" 05:39:46 INFO [loop_until]: OK (rc = 0) 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [loop_until]: OK (rc = 0) 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- status code --- 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- stdout --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- stdout --- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 INFO [http_cmd]: http status code OK 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17387m 8639Mi am-767f6b67b5-qcqz2 17403m 8707Mi am-767f6b67b5-vbltz 19043m 8648Mi ds-cts-0 18072m 10623Mi ds-cts-1 16516m 10737Mi ds-cts-2 18106m 10677Mi ds-idrepo-0 7594m 3719Mi ds-idrepo-1 7519m 3634Mi ds-idrepo-2 8027m 3534Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 14m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7855m 7366Mi 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6196Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7915m 26% 7394Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17300m 57% 11345Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3384Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18045m 60% 13770Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17734m 59% 11593Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19036m 63% 3006Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19368m 64% 10866Mi 9% gke-xlou-cdm-ds-0f601751-j488 8183m 27% 6852Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16537m 55% 15712Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7935m 26% 7526Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18415m 61% 13254Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8189m 51% 11193Mi 19% 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG --- status code --- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG --- stderr --- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- stderr --- 05:39:46 DEBUG 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:39:46 DEBUG { 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG --- http response --- 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG { 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:39:46 DEBUG { 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:39:46 DEBUG { 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG { 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG { 05:39:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:39:46 DEBUG "data": { 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG "data": { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG { 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG { 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG "data": { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG ----- output ----- 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "data": { 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG { 05:39:46 DEBUG "data": { 05:39:46 DEBUG "data": { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "data": { 05:39:46 DEBUG { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "data": { 05:39:46 DEBUG { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "data": { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "status": "success", 05:39:46 DEBUG { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "data": { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "data": { 05:39:46 DEBUG { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "data": { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG { 05:39:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:46 DEBUG { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "metric": { 05:39:46 DEBUG { 05:39:46 DEBUG "resultType": "vector", 05:39:46 DEBUG "pod": "ds-idrepo-0" 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG { 05:39:46 DEBUG { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG }, 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "result": [ 05:39:46 DEBUG }, 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:39:46 DEBUG "pod": "ds-idrepo-0" 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:46 DEBUG "instance": "10.142.0.107:9100" 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "pod": "ds-cts-0" 05:39:46 DEBUG { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG { 05:39:46 DEBUG }, 05:39:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:39:46 DEBUG { 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG }, 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:46 DEBUG }, 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "pod": "ds-cts-0" 05:39:46 DEBUG "metric": { 05:39:46 DEBUG "value": [ 05:39:46 DEBUG }, 05:39:46 DEBUG "metric": { 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "instance": "10.142.0.107:9100" 05:39:46 DEBUG "instance": "10.142.0.107:9100" 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:39:46 DEBUG }, 05:39:46 DEBUG "pod": "ds-cts-0" 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "pod": "ds-idrepo-0" 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "pod": "ds-idrepo-2" 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG [...] 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "pod": "ds-cts-2" 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG "pod": "ds-cts-2" 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:46 DEBUG "instance": "10.142.0.43:9100" 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "pod": "ds-cts-2" 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 05:39:46 DEBUG 05:39:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:39:46 DEBUG "pod": "ds-idrepo-2" 05:39:46 DEBUG "value": [ 05:39:46 DEBUG }, 05:39:46 DEBUG "instance": "10.142.0.43:9100" 05:39:46 DEBUG 05:39:46 DEBUG "instance": "10.142.0.43:9100" 05:39:46 DEBUG }, 05:39:46 DEBUG }, 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG "pod": "ds-cts-2" 05:39:46 DEBUG 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG 05:39:46 DEBUG "pod": "ds-cts-2" 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0" 05:39:46 DEBUG }, 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG }, 05:39:46 DEBUG 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "2673.5333333333333" 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:46 DEBUG }, 05:39:46 DEBUG "pod": "ds-cts-2" 05:39:46 DEBUG "18771.154576958583" 05:39:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:46 DEBUG }, 05:39:46 DEBUG ] 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "8.030514220664832" 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "34416693.64761429" 05:39:46 DEBUG "value": [ 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG ] 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG }, 05:39:46 DEBUG ] 05:39:46 DEBUG }, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG } 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG ] 05:39:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:39:46 DEBUG ] 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "480646.5" 05:39:46 DEBUG "80464684.69999999" 05:39:46 DEBUG } 05:39:46 DEBUG "0" 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "value": [ 05:39:46 DEBUG } 05:39:46 DEBUG "value": [ 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG ] 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0" 05:39:46 DEBUG } 05:39:46 DEBUG }, 05:39:46 DEBUG } 05:39:46 DEBUG "0.17523333333331834" 05:39:46 DEBUG "3052578483200" 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0" 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0.0007666666666666666" 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG "0" 05:39:46 DEBUG } 05:39:46 DEBUG "7822053376" 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG "value": [ 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG "2675.8" 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG "2672.9" 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG 1713587986, 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG "0.092" 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG ] 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:39:46 DEBUG } 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588046" 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:40:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "292431281.14820194" 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "2675.6666666666665" 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 ERROR [http_cmd]: ERROR 05:40:46 ERROR --- status code --- 05:40:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:40:46 ERROR --- http response --- 05:40:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:40:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588046" 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "instance": "10.142.0.107:9100" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "instance": "10.142.0.43:9100" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "80609296.76666667" 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "18.78168405095939" 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "instance": "10.142.0.107:9100" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "instance": "10.142.0.43:9100" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "0.06133333333332682" 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "ds-idrepo-0" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "ds-idrepo-2" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "0" 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "0" 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 INFO 05:40:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:40:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:46 INFO 05:40:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:40:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588046" 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [loop_until]: OK (rc = 0) 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [loop_until]: OK (rc = 0) 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- status code --- 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- stdout --- 05:40:46 DEBUG --- status code --- 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- stdout --- 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- status code --- 05:40:46 INFO [http_cmd]: http status code OK 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17744m 8643Mi am-767f6b67b5-qcqz2 16998m 8704Mi am-767f6b67b5-vbltz 18951m 8645Mi ds-cts-0 18324m 10624Mi ds-cts-1 16634m 10719Mi ds-cts-2 19163m 10664Mi ds-idrepo-0 7702m 3719Mi ds-idrepo-1 7437m 3658Mi ds-idrepo-2 8097m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 13m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8115m 7367Mi 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- status code --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- stderr --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6213Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7907m 26% 7391Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17305m 57% 11352Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3387Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17508m 58% 13747Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5215Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17721m 59% 11597Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18910m 63% 3007Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19387m 64% 10854Mi 9% gke-xlou-cdm-ds-0f601751-j488 8183m 27% 6864Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16949m 56% 15702Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7983m 26% 7540Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18830m 63% 13252Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8177m 51% 11190Mi 19% 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:46 DEBUG 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG --- stderr --- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:40:46 DEBUG --- http response --- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG ----- output ----- 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "data": { 05:40:46 DEBUG "data": { 05:40:46 DEBUG { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG { 05:40:46 DEBUG "data": { 05:40:46 DEBUG "data": { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "data": { 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "status": "success", 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "data": { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "data": { 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "resultType": "vector", 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "result": [ 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "result": [ 05:40:46 DEBUG "instance": "10.142.0.107:9100" 05:40:46 DEBUG { 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:46 DEBUG { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "ds-idrepo-0" 05:40:46 DEBUG { 05:40:46 DEBUG }, 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "ds-cts-0" 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:46 DEBUG "instance": "10.142.0.107:9100" 05:40:46 DEBUG }, 05:40:46 DEBUG "pod": "ds-idrepo-0" 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "pod": "ds-idrepo-0" 05:40:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:40:46 DEBUG }, 05:40:46 DEBUG "metric": { 05:40:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:40:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:40:46 DEBUG "instance": "10.142.0.107:9100" 05:40:46 DEBUG }, 05:40:46 DEBUG "pod": "ds-cts-0" 05:40:46 DEBUG }, 05:40:46 DEBUG "pod": "ds-cts-0" 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG [...] 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG [...] 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "ds-cts-2" 05:40:46 DEBUG "instance": "10.142.0.43:9100" 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG }, 05:40:46 DEBUG "pod": "ds-cts-2" 05:40:46 DEBUG 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG "instance": "10.142.0.43:9100" 05:40:46 DEBUG }, 05:40:46 DEBUG "pod": "ds-idrepo-2" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG "pod": "ds-idrepo-2" 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:40:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:40:46 DEBUG "instance": "10.142.0.43:9100" 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 05:40:46 DEBUG }, 05:40:46 DEBUG "pod": "ds-cts-2" 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "pod": "ds-cts-2" 05:40:46 DEBUG }, 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "2668.4666666666667" 05:40:46 DEBUG }, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "0.092" 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "2668.9" 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG ] 05:40:46 DEBUG "value": [ 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "0" 05:40:46 DEBUG "0.1716333333333599" 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG ] 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "0" 05:40:46 DEBUG "80604507.13333333" 05:40:46 DEBUG ] 05:40:46 DEBUG "0" 05:40:46 DEBUG "480828.5" 05:40:46 DEBUG } 05:40:46 DEBUG 1713588046, 05:40:46 DEBUG "0" 05:40:46 DEBUG "0" 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG "8.030397313583391" 05:40:46 DEBUG "7823114240" 05:40:46 DEBUG "3056012096512" 05:40:46 DEBUG } 05:40:46 DEBUG "0.0016333333333333334" 05:40:46 DEBUG "0" 05:40:46 DEBUG "1056.8617964844382" 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG "2668.7666666666664" 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG ] 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:40:46 DEBUG } 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 ERROR [http_cmd]: ERROR 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 ERROR --- status code --- 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 ERROR --- http response --- 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "44848482.50904705" 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "2670.6666666666665" 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588106" 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "instance": "10.142.0.107:9100" 05:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:41:46 DEBUG "instance": "10.142.0.43:9100" 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "80775087.13333333" 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG ] 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG "0.0032216784421814047" 05:41:46 DEBUG } 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:46 DEBUG } 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG { 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "data": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "instance": "10.142.0.107:9100" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "instance": "10.142.0.43:9100" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "0.10400000000000391" 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG { 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "data": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "pod": "ds-idrepo-0" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "pod": "ds-idrepo-2" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "0" 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG { 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "data": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "0" 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 INFO 05:41:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:46 INFO 05:41:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588106" 05:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588106" 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [loop_until]: OK (rc = 0) 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [loop_until]: OK (rc = 0) 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- status code --- 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- stdout --- 05:41:46 DEBUG --- status code --- 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- stdout --- 05:41:46 DEBUG --- status code --- 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17212m 8644Mi am-767f6b67b5-qcqz2 17387m 8704Mi am-767f6b67b5-vbltz 18838m 8645Mi ds-cts-0 18268m 10622Mi ds-cts-1 17187m 10712Mi ds-cts-2 18917m 10658Mi ds-idrepo-0 7826m 3719Mi ds-idrepo-1 7684m 3654Mi ds-idrepo-2 7908m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-799cdc5bdb-7t5x7 14m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8020m 7367Mi 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6213Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7926m 26% 7384Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17301m 57% 11348Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3382Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18403m 61% 13742Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17601m 58% 11593Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18755m 62% 2997Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19432m 65% 10860Mi 9% gke-xlou-cdm-ds-0f601751-j488 8149m 27% 6851Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17158m 57% 15698Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7990m 26% 7548Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18808m 62% 13254Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8192m 51% 11203Mi 19% 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- stderr --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- http response --- 05:41:46 INFO [http_cmd]: http status code OK 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:46 DEBUG --- stderr --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:46 DEBUG --- status code --- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:41:46 DEBUG { 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG { 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG --- http response --- 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG ----- output ----- 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "data": { 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "status": "success", 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "data": { 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "data": { 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "result": [ 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG { 05:41:46 DEBUG "data": { 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "pod": "ds-cts-0" 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG { 05:41:46 DEBUG { 05:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "instance": "10.142.0.107:9100" 05:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:46 DEBUG { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG { 05:41:46 DEBUG 05:41:46 DEBUG "resultType": "vector", 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "metric": { 05:41:46 DEBUG }, 05:41:46 DEBUG "pod": "ds-idrepo-0" 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:46 DEBUG "pod": "ds-idrepo-0" 05:41:46 DEBUG { 05:41:46 DEBUG "instance": "10.142.0.107:9100" 05:41:46 DEBUG "pod": "ds-cts-0" 05:41:46 DEBUG "pod": "ds-cts-0" 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:46 DEBUG "metric": { 05:41:46 DEBUG 05:41:46 DEBUG "result": [ 05:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG }, 05:41:46 DEBUG { 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG "metric": { 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG }, 05:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:41:46 DEBUG }, 05:41:46 DEBUG "instance": "10.142.0.107:9100" 05:41:46 DEBUG 05:41:46 DEBUG { 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG }, 05:41:46 DEBUG [...] 05:41:46 DEBUG "metric": { 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG "pod": "ds-idrepo-0" 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG "pod": "ds-cts-2" 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "pod": "ds-cts-2" 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:41:46 DEBUG 05:41:46 DEBUG [...] 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG "pod": "ds-cts-2" 05:41:46 DEBUG 05:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:46 DEBUG "pod": "ds-idrepo-2" 05:41:46 DEBUG 05:41:46 DEBUG "instance": "10.142.0.43:9100" 05:41:46 DEBUG "pod": "ds-cts-2" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:46 DEBUG 05:41:46 DEBUG "0" 05:41:46 DEBUG 05:41:46 DEBUG 05:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG 05:41:46 DEBUG }, 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "instance": "10.142.0.43:9100" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:41:46 DEBUG }, 05:41:46 DEBUG "instance": "10.142.0.43:9100" 05:41:46 DEBUG ] 05:41:46 DEBUG 05:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG }, 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG }, 05:41:46 DEBUG } 05:41:46 DEBUG "pod": "ds-idrepo-2" 05:41:46 DEBUG }, 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG }, 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "0.0008584866666666669" 05:41:46 DEBUG "0" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "2658.0666666666666" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG ] 05:41:46 DEBUG }, 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "2658.133333333333" 05:41:46 DEBUG "564.6734447699466" 05:41:46 DEBUG "0" 05:41:46 DEBUG }, 05:41:46 DEBUG "480619.5" 05:41:46 DEBUG "0" 05:41:46 DEBUG "value": [ 05:41:46 DEBUG "80770254.76666667" 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG ] 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "2658.0666666666666" 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG } 05:41:46 DEBUG "value": [ 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG "0.092" 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG "value": [ 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG "0.2511333333333208" 05:41:46 DEBUG } 05:41:46 DEBUG "0" 05:41:46 DEBUG ] 05:41:46 DEBUG "3059970810880" 05:41:46 DEBUG } 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG 1713588106, 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG "0" 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG "7823835136" 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG "8.030889813949244" 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG ] 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:41:46 DEBUG } 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588166" 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588166" 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588166" 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588166" 05:42:46 ERROR [http_cmd]: ERROR 05:42:46 ERROR --- status code --- 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- status code --- 05:42:46 ERROR --- http response --- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- http response --- 05:42:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:42:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:42:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 DEBUG --- status code --- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- http response --- 05:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:42:46 DEBUG ----- output ----- 05:42:46 DEBUG { 05:42:46 DEBUG "status": "success", 05:42:46 DEBUG "data": { 05:42:46 DEBUG "resultType": "vector", 05:42:46 DEBUG "result": [ 05:42:46 DEBUG { 05:42:46 DEBUG "metric": { 05:42:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG [...] 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 1713588166, 05:42:46 DEBUG "0" 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG } 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 DEBUG --- status code --- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- http response --- 05:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:46 DEBUG ----- output ----- 05:42:46 DEBUG { 05:42:46 DEBUG "status": "success", 05:42:46 DEBUG "data": { 05:42:46 DEBUG "resultType": "vector", 05:42:46 DEBUG "result": [ 05:42:46 DEBUG { 05:42:46 DEBUG "metric": { 05:42:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG [...] 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 1713588166, 05:42:46 DEBUG "2677.5666666666666" 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG } 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588166" 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588166" 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588166" 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 DEBUG --- status code --- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- http response --- 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 DEBUG --- status code --- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- http response --- 05:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 DEBUG ----- output ----- 05:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:46 DEBUG --- status code --- 05:42:46 DEBUG { 05:42:46 DEBUG ----- output ----- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG "status": "success", 05:42:46 DEBUG { 05:42:46 DEBUG --- http response --- 05:42:46 DEBUG "data": { 05:42:46 DEBUG "status": "success", 05:42:46 DEBUG "resultType": "vector", 05:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:46 DEBUG "data": { 05:42:46 DEBUG "result": [ 05:42:46 DEBUG ----- output ----- 05:42:46 DEBUG "resultType": "vector", 05:42:46 DEBUG { 05:42:46 DEBUG { 05:42:46 DEBUG "result": [ 05:42:46 DEBUG "metric": { 05:42:46 DEBUG "status": "success", 05:42:46 DEBUG { 05:42:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:46 DEBUG "data": { 05:42:46 DEBUG "metric": { 05:42:46 DEBUG }, 05:42:46 DEBUG "resultType": "vector", 05:42:46 DEBUG "instance": "10.142.0.107:9100" 05:42:46 DEBUG "value": [ 05:42:46 DEBUG "result": [ 05:42:46 DEBUG }, 05:42:46 DEBUG 05:42:46 DEBUG { 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 05:42:46 DEBUG "metric": { 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG "instance": "10.142.0.107:9100" 05:42:46 DEBUG 05:42:46 DEBUG [...] 05:42:46 DEBUG }, 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG "value": [ 05:42:46 DEBUG [...] 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG }, 05:42:46 DEBUG [...] 05:42:46 DEBUG "instance": "10.142.0.43:9100" 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 05:42:46 DEBUG }, 05:42:46 DEBUG 1713588166, 05:42:46 DEBUG 05:42:46 DEBUG "value": [ 05:42:46 DEBUG "0" 05:42:46 DEBUG 05:42:46 DEBUG 1713588166, 05:42:46 DEBUG ] 05:42:46 DEBUG "instance": "10.142.0.43:9100" 05:42:46 DEBUG "80762926.69999999" 05:42:46 DEBUG } 05:42:46 DEBUG }, 05:42:46 DEBUG ] 05:42:46 DEBUG ] 05:42:46 DEBUG "value": [ 05:42:46 DEBUG } 05:42:46 DEBUG } 05:42:46 DEBUG 1713588166, 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG "0.07499999999999053" 05:42:46 DEBUG } 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG } 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG } 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588166" 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 DEBUG --- status code --- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- http response --- 05:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:46 DEBUG ----- output ----- 05:42:46 DEBUG { 05:42:46 DEBUG "status": "success", 05:42:46 DEBUG "data": { 05:42:46 DEBUG "resultType": "vector", 05:42:46 DEBUG "result": [ 05:42:46 DEBUG { 05:42:46 DEBUG "metric": { 05:42:46 DEBUG "pod": "ds-idrepo-0" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG [...] 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG "pod": "ds-idrepo-2" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 1713588166, 05:42:46 DEBUG "0" 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG } 05:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588166" 05:42:46 INFO [http_cmd]: http status code OK 05:42:46 DEBUG --- status code --- 05:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:46 DEBUG --- http response --- 05:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:46 DEBUG ----- output ----- 05:42:46 DEBUG { 05:42:46 DEBUG "status": "success", 05:42:46 DEBUG "data": { 05:42:46 DEBUG "resultType": "vector", 05:42:46 DEBUG "result": [ 05:42:46 DEBUG { 05:42:46 DEBUG "metric": { 05:42:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG [...] 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG 05:42:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:46 DEBUG }, 05:42:46 DEBUG "value": [ 05:42:46 DEBUG 1713588166, 05:42:46 DEBUG "0" 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG ] 05:42:46 DEBUG } 05:42:46 DEBUG } 05:42:46 INFO 05:42:46 INFO [loop_until]: kubectl --namespace=xlou top pods 05:42:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:46 INFO 05:42:46 INFO [loop_until]: kubectl --namespace=xlou top node 05:42:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588167" 05:42:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588167" 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [loop_until]: OK (rc = 0) 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 INFO [loop_until]: OK (rc = 0) 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG --- stdout --- 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- stdout --- 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 INFO [http_cmd]: http status code OK 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17149m 8644Mi am-767f6b67b5-qcqz2 17348m 8705Mi am-767f6b67b5-vbltz 18695m 8649Mi ds-cts-0 18744m 10598Mi ds-cts-1 17424m 10713Mi ds-cts-2 19515m 10629Mi ds-idrepo-0 7696m 3719Mi ds-idrepo-1 7501m 3666Mi ds-idrepo-2 7979m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7991m 7376Mi 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6205Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7981m 26% 7384Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17353m 58% 11351Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3378Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18957m 63% 13764Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5210Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17584m 58% 11590Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18979m 63% 2995Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19480m 65% 10859Mi 9% gke-xlou-cdm-ds-0f601751-j488 8173m 27% 6849Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17310m 57% 15676Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7987m 26% 7547Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19526m 65% 13264Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8187m 51% 11195Mi 19% 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG --- status code --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- stderr --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- stderr --- 05:42:47 DEBUG 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG "data": { 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG --- http response --- 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:42:47 DEBUG { 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG { 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG { 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG "data": { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "data": { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG ----- output ----- 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "data": { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "data": { 05:42:47 DEBUG { 05:42:47 DEBUG "data": { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG { 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "status": "success", 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "data": { 05:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:47 DEBUG { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "data": { 05:42:47 DEBUG "data": { 05:42:47 DEBUG { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "metric": { 05:42:47 DEBUG { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "data": { 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG "instance": "10.142.0.107:9100" 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "metric": { 05:42:47 DEBUG { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG { 05:42:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "instance": "10.142.0.107:9100" 05:42:47 DEBUG "metric": { 05:42:47 DEBUG { 05:42:47 DEBUG "resultType": "vector", 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "pod": "ds-cts-0" 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "pod": "ds-cts-0" 05:42:47 DEBUG "metric": { 05:42:47 DEBUG }, 05:42:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:47 DEBUG }, 05:42:47 DEBUG "pod": "ds-idrepo-0" 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "result": [ 05:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:47 DEBUG }, 05:42:47 DEBUG 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "value": [ 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG { 05:42:47 DEBUG }, 05:42:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:42:47 DEBUG }, 05:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:47 DEBUG "value": [ 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG }, 05:42:47 DEBUG "pod": "ds-idrepo-0" 05:42:47 DEBUG { 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG "instance": "10.142.0.107:9100" 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "value": [ 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG }, 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG }, 05:42:47 DEBUG "metric": { 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG }, 05:42:47 DEBUG 05:42:47 DEBUG "metric": { 05:42:47 DEBUG "pod": "ds-cts-0" 05:42:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG "pod": "ds-idrepo-0" 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG [...] 05:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:42:47 DEBUG }, 05:42:47 DEBUG }, 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG [...] 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG "instance": "10.142.0.43:9100" 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG "instance": "10.142.0.43:9100" 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:47 DEBUG }, 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:47 DEBUG }, 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG "pod": "ds-cts-2" 05:42:47 DEBUG "value": [ 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG [...] 05:42:47 DEBUG 05:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:47 DEBUG 05:42:47 DEBUG "pod": "ds-cts-2" 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG 05:42:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG "pod": "ds-cts-2" 05:42:47 DEBUG }, 05:42:47 DEBUG "pod": "ds-cts-2" 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG "3064656897024" 05:42:47 DEBUG 05:42:47 DEBUG }, 05:42:47 DEBUG 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG "2665.0666666666666" 05:42:47 DEBUG 05:42:47 DEBUG "value": [ 05:42:47 DEBUG "80758424.76666667" 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG 05:42:47 DEBUG "pod": "ds-cts-2" 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG }, 05:42:47 DEBUG "2668.9333333333334" 05:42:47 DEBUG ] 05:42:47 DEBUG "pod": "ds-cts-2" 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 05:42:47 DEBUG "2669.733333333333" 05:42:47 DEBUG ] 05:42:47 DEBUG "pod": "ds-idrepo-2" 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG ] 05:42:47 DEBUG 05:42:47 DEBUG "instance": "10.142.0.43:9100" 05:42:47 DEBUG 05:42:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:42:47 DEBUG "480280" 05:42:47 DEBUG }, 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG }, 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG "pod": "ds-idrepo-2" 05:42:47 DEBUG } 05:42:47 DEBUG }, 05:42:47 DEBUG ] 05:42:47 DEBUG "0" 05:42:47 DEBUG } 05:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:42:47 DEBUG }, 05:42:47 DEBUG }, 05:42:47 DEBUG ] 05:42:47 DEBUG "value": [ 05:42:47 DEBUG "0" 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG "7829614592" 05:42:47 DEBUG }, 05:42:47 DEBUG ] 05:42:47 DEBUG "value": [ 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG ] 05:42:47 DEBUG }, 05:42:47 DEBUG "value": [ 05:42:47 DEBUG "value": [ 05:42:47 DEBUG } 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG } 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG ] 05:42:47 DEBUG "value": [ 05:42:47 DEBUG } 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG "value": [ 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG ] 05:42:47 DEBUG "25808940.020773225" 05:42:47 DEBUG ] 05:42:47 DEBUG "44911986.438102" 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG "0" 05:42:47 DEBUG } 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG } 05:42:47 DEBUG "0" 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG 1713588167, 05:42:47 DEBUG "0.17786666666677223" 05:42:47 DEBUG "8.047416451501332" 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG "0.00006666666666666662" 05:42:47 DEBUG ] 05:42:47 DEBUG ] 05:42:47 DEBUG "0" 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG "0.092" 05:42:47 DEBUG ] 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG ] 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:42:47 DEBUG } 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588226" 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588226" 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588226" 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588226" 05:43:46 ERROR [http_cmd]: ERROR 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 ERROR --- status code --- 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 DEBUG --- status code --- 05:43:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:43:46 DEBUG --- status code --- 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 ERROR --- http response --- 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:43:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:43:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:43:46 DEBUG ----- output ----- 05:43:46 DEBUG { 05:43:46 DEBUG "status": "success", 05:43:46 DEBUG "data": { 05:43:46 DEBUG "resultType": "vector", 05:43:46 DEBUG "result": [ 05:43:46 DEBUG { 05:43:46 DEBUG "metric": { 05:43:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG [...] 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG "pod": "ds-cts-2" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 DEBUG 1713588226, 05:43:46 DEBUG --- status code --- 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 DEBUG "0" 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG ] 05:43:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:46 DEBUG } 05:43:46 DEBUG ----- output ----- 05:43:46 DEBUG ] 05:43:46 DEBUG { 05:43:46 DEBUG } 05:43:46 DEBUG "status": "success", 05:43:46 DEBUG } 05:43:46 DEBUG "data": { 05:43:46 DEBUG "resultType": "vector", 05:43:46 DEBUG "result": [ 05:43:46 DEBUG { 05:43:46 DEBUG "metric": { 05:43:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG [...] 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 1713588226, 05:43:46 DEBUG "2669.733333333333" 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG } 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588226" 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588226" 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588226" 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 DEBUG --- status code --- 05:43:46 DEBUG --- status code --- 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:46 DEBUG ----- output ----- 05:43:46 DEBUG --- status code --- 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:46 DEBUG ----- output ----- 05:43:46 DEBUG { 05:43:46 DEBUG "status": "success", 05:43:46 DEBUG "data": { 05:43:46 DEBUG "resultType": "vector", 05:43:46 DEBUG "result": [ 05:43:46 DEBUG { 05:43:46 DEBUG "metric": { 05:43:46 DEBUG "instance": "10.142.0.107:9100" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG [...] 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG "instance": "10.142.0.43:9100" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 1713588226, 05:43:46 DEBUG "0.06700000000000537" 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG } 05:43:46 DEBUG { 05:43:46 DEBUG "status": "success", 05:43:46 DEBUG "data": { 05:43:46 DEBUG "resultType": "vector", 05:43:46 DEBUG "result": [ 05:43:46 DEBUG { 05:43:46 DEBUG "metric": { 05:43:46 DEBUG "instance": "10.142.0.107:9100" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG [...] 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG "instance": "10.142.0.43:9100" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 1713588226, 05:43:46 DEBUG "80767977.93333334" 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG } 05:43:46 DEBUG ----- output ----- 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588226" 05:43:46 DEBUG { 05:43:46 DEBUG "status": "success", 05:43:46 DEBUG "data": { 05:43:46 DEBUG "resultType": "vector", 05:43:46 DEBUG "result": [ 05:43:46 DEBUG { 05:43:46 DEBUG "metric": { 05:43:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG [...] 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG "pod": "ds-cts-2" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 1713588226, 05:43:46 DEBUG "0" 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG } 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 DEBUG --- status code --- 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:46 DEBUG ----- output ----- 05:43:46 DEBUG { 05:43:46 DEBUG "status": "success", 05:43:46 DEBUG "data": { 05:43:46 DEBUG "resultType": "vector", 05:43:46 DEBUG "result": [ 05:43:46 DEBUG { 05:43:46 DEBUG "metric": { 05:43:46 DEBUG "pod": "ds-idrepo-0" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG [...] 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG "pod": "ds-idrepo-2" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 1713588226, 05:43:46 DEBUG "0" 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG } 05:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588226" 05:43:46 INFO [http_cmd]: http status code OK 05:43:46 DEBUG --- status code --- 05:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:46 DEBUG --- http response --- 05:43:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:46 DEBUG ----- output ----- 05:43:46 DEBUG { 05:43:46 DEBUG "status": "success", 05:43:46 DEBUG "data": { 05:43:46 DEBUG "resultType": "vector", 05:43:46 DEBUG "result": [ 05:43:46 DEBUG { 05:43:46 DEBUG "metric": { 05:43:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG [...] 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG 05:43:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:46 DEBUG }, 05:43:46 DEBUG "value": [ 05:43:46 DEBUG 1713588226, 05:43:46 DEBUG "0" 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG ] 05:43:46 DEBUG } 05:43:46 DEBUG } 05:43:47 INFO 05:43:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:43:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:47 INFO 05:43:47 INFO [loop_until]: kubectl --namespace=xlou top pods 05:43:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588227" 05:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588227" 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [loop_until]: OK (rc = 0) 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 INFO [loop_until]: OK (rc = 0) 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG --- stdout --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- stdout --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 INFO [http_cmd]: http status code OK 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6206Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7942m 26% 7381Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17412m 58% 11355Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3387Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18317m 61% 13759Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5229Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17611m 58% 11589Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18785m 62% 2987Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 18792m 62% 10864Mi 9% gke-xlou-cdm-ds-0f601751-j488 8202m 27% 6855Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16217m 54% 15703Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7952m 26% 7563Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19301m 64% 13255Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8182m 51% 11197Mi 19% 05:43:47 DEBUG --- stderr --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- status code --- 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG { 05:43:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17648m 8646Mi am-767f6b67b5-qcqz2 17984m 8712Mi am-767f6b67b5-vbltz 18965m 8649Mi ds-cts-0 18309m 10624Mi ds-cts-1 16577m 10714Mi ds-cts-2 19162m 10653Mi ds-idrepo-0 7866m 3719Mi ds-idrepo-1 7583m 3665Mi ds-idrepo-2 8040m 3535Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 19m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7954m 7377Mi 05:43:47 DEBUG 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG --- stderr --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG { 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG --- http response --- 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG { 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:43:47 DEBUG { 05:43:47 DEBUG { 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG { 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG "data": { 05:43:47 DEBUG "data": { 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "instance": "10.142.0.107:9100" 05:43:47 DEBUG "data": { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "instance": "10.142.0.107:9100" 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "data": { 05:43:47 DEBUG ----- output ----- 05:43:47 DEBUG { 05:43:47 DEBUG }, 05:43:47 DEBUG "data": { 05:43:47 DEBUG { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "data": { 05:43:47 DEBUG }, 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG { 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "data": { 05:43:47 DEBUG { 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "status": "success", 05:43:47 DEBUG "data": { 05:43:47 DEBUG 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:47 DEBUG { 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG }, 05:43:47 DEBUG "data": { 05:43:47 DEBUG "value": [ 05:43:47 DEBUG { 05:43:47 DEBUG "instance": "10.142.0.107:9100" 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "metric": { 05:43:47 DEBUG { 05:43:47 DEBUG "data": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG 05:43:47 DEBUG { 05:43:47 DEBUG }, 05:43:47 DEBUG "metric": { 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "result": [ 05:43:47 DEBUG "pod": "ds-idrepo-0" 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "resultType": "vector", 05:43:47 DEBUG "result": [ 05:43:47 DEBUG 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "pod": "ds-idrepo-0" 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG "result": [ 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG }, 05:43:47 DEBUG { 05:43:47 DEBUG { 05:43:47 DEBUG }, 05:43:47 DEBUG "pod": "ds-cts-0" 05:43:47 DEBUG "result": [ 05:43:47 DEBUG { 05:43:47 DEBUG [...] 05:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG { 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "value": [ 05:43:47 DEBUG }, 05:43:47 DEBUG { 05:43:47 DEBUG "metric": { 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "metric": { 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "pod": "ds-idrepo-0" 05:43:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "metric": { 05:43:47 DEBUG "pod": "ds-cts-0" 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG "pod": "ds-cts-0" 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "instance": "10.142.0.43:9100" 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG [...] 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "instance": "10.142.0.43:9100" 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG "instance": "10.142.0.43:9100" 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG [...] 05:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "value": [ 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "0.17380000000001036" 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "pod": "ds-cts-2" 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG ] 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:47 DEBUG }, 05:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG [...] 05:43:47 DEBUG 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG 05:43:47 DEBUG } 05:43:47 DEBUG 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG "7831281664" 05:43:47 DEBUG "3068120761344" 05:43:47 DEBUG 05:43:47 DEBUG "80763751" 05:43:47 DEBUG 05:43:47 DEBUG "pod": "ds-cts-2" 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "pod": "ds-idrepo-2" 05:43:47 DEBUG 05:43:47 DEBUG "pod": "ds-cts-2" 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "0" 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG }, 05:43:47 DEBUG 05:43:47 DEBUG }, 05:43:47 DEBUG "2669.1" 05:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:43:47 DEBUG "2668.2666666666664" 05:43:47 DEBUG ] 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG "pod": "ds-cts-2" 05:43:47 DEBUG } 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "value": [ 05:43:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:43:47 DEBUG "value": [ 05:43:47 DEBUG ] 05:43:47 DEBUG }, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG }, 05:43:47 DEBUG "2667.4666666666667" 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG }, 05:43:47 DEBUG "0.092" 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG "pod": "ds-idrepo-2" 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG }, 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG "value": [ 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG "value": [ 05:43:47 DEBUG ] 05:43:47 DEBUG "0" 05:43:47 DEBUG }, 05:43:47 DEBUG "0" 05:43:47 DEBUG "0" 05:43:47 DEBUG } 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG "8.045086338664584" 05:43:47 DEBUG ] 05:43:47 DEBUG "480432" 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG "value": [ 05:43:47 DEBUG 1713588227, 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG "426.4388122856934" 05:43:47 DEBUG "0" 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG "0.0004666666666666666" 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG "0" 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG ] 05:43:47 DEBUG ] 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:43:47 DEBUG } 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588286" 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588286" 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588286" 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588286" 05:44:46 ERROR [http_cmd]: ERROR 05:44:46 ERROR --- status code --- 05:44:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:44:46 ERROR --- http response --- 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:44:46 DEBUG --- status code --- 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 DEBUG --- status code --- 05:44:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG --- status code --- 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:44:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:44:46 DEBUG ----- output ----- 05:44:46 DEBUG ----- output ----- 05:44:46 DEBUG { 05:44:46 DEBUG { 05:44:46 DEBUG "status": "success", 05:44:46 DEBUG "status": "success", 05:44:46 DEBUG "data": { 05:44:46 DEBUG "data": { 05:44:46 DEBUG "resultType": "vector", 05:44:46 DEBUG "resultType": "vector", 05:44:46 DEBUG "result": [ 05:44:46 DEBUG "result": [ 05:44:46 DEBUG { 05:44:46 DEBUG { 05:44:46 DEBUG "metric": { 05:44:46 DEBUG "metric": { 05:44:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:46 DEBUG }, 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG [...] 05:44:46 DEBUG [...] 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:46 DEBUG "pod": "ds-cts-0" 05:44:46 DEBUG }, 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 1713588286, 05:44:46 DEBUG 1713588286, 05:44:46 DEBUG "2658.9666666666667" 05:44:46 DEBUG "364006067.0544409" 05:44:46 DEBUG ] 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 DEBUG ] 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588286" 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588286" 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588286" 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588286" 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 DEBUG --- status code --- 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:46 DEBUG ----- output ----- 05:44:46 DEBUG { 05:44:46 DEBUG "status": "success", 05:44:46 DEBUG "data": { 05:44:46 DEBUG "resultType": "vector", 05:44:46 DEBUG "result": [ 05:44:46 DEBUG { 05:44:46 DEBUG "metric": { 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 DEBUG "instance": "10.142.0.107:9100" 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 DEBUG --- status code --- 05:44:46 DEBUG }, 05:44:46 DEBUG --- status code --- 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG "value": [ 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:46 DEBUG 05:44:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:44:46 DEBUG ----- output ----- 05:44:46 DEBUG 05:44:46 DEBUG ----- output ----- 05:44:46 DEBUG { 05:44:46 DEBUG [...] 05:44:46 DEBUG { 05:44:46 DEBUG "status": "success", 05:44:46 DEBUG 05:44:46 DEBUG "status": "success", 05:44:46 DEBUG "data": { 05:44:46 DEBUG 05:44:46 DEBUG "data": { 05:44:46 DEBUG "resultType": "vector", 05:44:46 DEBUG 05:44:46 DEBUG "resultType": "vector", 05:44:46 DEBUG "result": [ 05:44:46 DEBUG "instance": "10.142.0.43:9100" 05:44:46 DEBUG "result": [ 05:44:46 DEBUG { 05:44:46 DEBUG }, 05:44:46 DEBUG { 05:44:46 DEBUG "metric": { 05:44:46 DEBUG "value": [ 05:44:46 DEBUG "metric": { 05:44:46 DEBUG "instance": "10.142.0.107:9100" 05:44:46 DEBUG 1713588286, 05:44:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:46 DEBUG }, 05:44:46 DEBUG "80857030.83333334" 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG ] 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 05:44:46 DEBUG } 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG ] 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG } 05:44:46 DEBUG 05:44:46 DEBUG [...] 05:44:46 DEBUG } 05:44:46 DEBUG [...] 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG "instance": "10.142.0.43:9100" 05:44:46 DEBUG "pod": "ds-cts-0" 05:44:46 DEBUG }, 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 1713588286, 05:44:46 DEBUG 1713588286, 05:44:46 DEBUG "0.10400000000000581" 05:44:46 DEBUG "18.505420528068957" 05:44:46 DEBUG ] 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 DEBUG ] 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 DEBUG --- status code --- 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:46 DEBUG ----- output ----- 05:44:46 DEBUG { 05:44:46 DEBUG "status": "success", 05:44:46 DEBUG "data": { 05:44:46 DEBUG "resultType": "vector", 05:44:46 DEBUG "result": [ 05:44:46 DEBUG { 05:44:46 DEBUG "metric": { 05:44:46 DEBUG "pod": "ds-idrepo-0" 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG [...] 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG "pod": "ds-idrepo-2" 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 1713588286, 05:44:46 DEBUG "0" 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588286" 05:44:46 INFO [http_cmd]: http status code OK 05:44:46 DEBUG --- status code --- 05:44:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:46 DEBUG --- http response --- 05:44:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:46 DEBUG ----- output ----- 05:44:46 DEBUG { 05:44:46 DEBUG "status": "success", 05:44:46 DEBUG "data": { 05:44:46 DEBUG "resultType": "vector", 05:44:46 DEBUG "result": [ 05:44:46 DEBUG { 05:44:46 DEBUG "metric": { 05:44:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG [...] 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG 05:44:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:46 DEBUG }, 05:44:46 DEBUG "value": [ 05:44:46 DEBUG 1713588286, 05:44:46 DEBUG "0" 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG ] 05:44:46 DEBUG } 05:44:46 DEBUG } 05:44:47 INFO 05:44:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:44:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:47 INFO 05:44:47 INFO [loop_until]: kubectl --namespace=xlou top pods 05:44:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 DEBUG --- http response --- 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588287" 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 INFO [loop_until]: OK (rc = 0) 05:44:47 INFO [loop_until]: OK (rc = 0) 05:44:47 DEBUG --- status code --- 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG ----- output ----- 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- status code --- 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- stdout --- 05:44:47 DEBUG --- stdout --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG { 05:44:47 DEBUG --- status code --- 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 INFO [http_cmd]: http status code OK 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6207Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7941m 26% 7384Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17421m 58% 11355Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3394Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18485m 61% 13767Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5226Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17044m 57% 11598Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18781m 62% 3000Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19491m 65% 10863Mi 9% gke-xlou-cdm-ds-0f601751-j488 8204m 27% 6852Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17008m 56% 15708Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7979m 26% 7561Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19133m 64% 13239Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8175m 51% 11194Mi 19% 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17429m 8648Mi am-767f6b67b5-qcqz2 17578m 8709Mi am-767f6b67b5-vbltz 18939m 8647Mi ds-cts-0 18475m 10622Mi ds-cts-1 17199m 10714Mi ds-cts-2 18847m 10654Mi ds-idrepo-0 7754m 3719Mi ds-idrepo-1 7573m 3669Mi ds-idrepo-2 7951m 3534Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 14m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7844m 7377Mi 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG --- status code --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- stderr --- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:47 DEBUG --- stderr --- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG "data": { 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "result": [ 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG --- http response --- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG { 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG ----- output ----- 05:44:47 DEBUG { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "instance": "10.142.0.107:9100" 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "data": { 05:44:47 DEBUG }, 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "status": "success", 05:44:47 DEBUG "data": { 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "data": { 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "data": { 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "result": [ 05:44:47 DEBUG 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "result": [ 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG { 05:44:47 DEBUG 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "resultType": "vector", 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:47 DEBUG "metric": { 05:44:47 DEBUG { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:47 DEBUG { 05:44:47 DEBUG { 05:44:47 DEBUG "pod": "ds-idrepo-0" 05:44:47 DEBUG }, 05:44:47 DEBUG "metric": { 05:44:47 DEBUG 05:44:47 DEBUG { 05:44:47 DEBUG "result": [ 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:44:47 DEBUG }, 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:47 DEBUG }, 05:44:47 DEBUG "pod": "ds-cts-0" 05:44:47 DEBUG }, 05:44:47 DEBUG { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "instance": "10.142.0.107:9100" 05:44:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:47 DEBUG }, 05:44:47 DEBUG "metric": { 05:44:47 DEBUG { 05:44:47 DEBUG "pod": "ds-idrepo-0" 05:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "instance": "10.142.0.107:9100" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "metric": { 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "metric": { 05:44:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:44:47 DEBUG "pod": "ds-idrepo-0" 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG [...] 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "pod": "ds-cts-0" 05:44:47 DEBUG "metric": { 05:44:47 DEBUG }, 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:44:47 DEBUG }, 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG "pod": "ds-cts-0" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "instance": "10.142.0.43:9100" 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:44:47 DEBUG 05:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "pod": "ds-cts-2" 05:44:47 DEBUG 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG [...] 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG "0.2670999999999869" 05:44:47 DEBUG "pod": "ds-cts-0" 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:44:47 DEBUG 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "instance": "10.142.0.43:9100" 05:44:47 DEBUG ] 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG 05:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:47 DEBUG 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG }, 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG 05:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 05:44:47 DEBUG }, 05:44:47 DEBUG 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG }, 05:44:47 DEBUG } 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "pod": "ds-cts-2" 05:44:47 DEBUG "pod": "ds-cts-2" 05:44:47 DEBUG "pod": "ds-idrepo-2" 05:44:47 DEBUG }, 05:44:47 DEBUG "instance": "10.142.0.43:9100" 05:44:47 DEBUG "7830949888" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "2669.4" 05:44:47 DEBUG "pod": "ds-cts-2" 05:44:47 DEBUG }, 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG "pod": "ds-cts-0" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "pod": "ds-idrepo-2" 05:44:47 DEBUG "0" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG ] 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG }, 05:44:47 DEBUG }, 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG }, 05:44:47 DEBUG ] 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG ] 05:44:47 DEBUG }, 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "2668.2666666666664" 05:44:47 DEBUG }, 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG }, 05:44:47 DEBUG ] 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG } 05:44:47 DEBUG "32713390.23593229" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG } 05:44:47 DEBUG "2668.7" 05:44:47 DEBUG } 05:44:47 DEBUG "value": [ 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG ] 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "8.03751348747923" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG } 05:44:47 DEBUG "80853762.66666667" 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG "0" 05:44:47 DEBUG "value": [ 05:44:47 DEBUG "480229.5" 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG "0.092" 05:44:47 DEBUG } 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG ] 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG "0" 05:44:47 DEBUG "0.00030917555555555557" 05:44:47 DEBUG ] 05:44:47 DEBUG 1713588287, 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG "0" 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG "0" 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG "65131780.6679302" 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG "3071801107456" 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG ] 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:44:47 DEBUG } 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588346" 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588346" 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588346" 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588346" 05:45:46 ERROR [http_cmd]: ERROR 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 ERROR --- status code --- 05:45:46 DEBUG --- status code --- 05:45:46 DEBUG --- status code --- 05:45:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 ERROR --- http response --- 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 DEBUG --- http response --- 05:45:46 DEBUG --- http response --- 05:45:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:45:46 DEBUG --- status code --- 05:45:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:45:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:45:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 DEBUG ----- output ----- 05:45:46 DEBUG ----- output ----- 05:45:46 DEBUG --- http response --- 05:45:46 DEBUG { 05:45:46 DEBUG { 05:45:46 DEBUG "status": "success", 05:45:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:45:46 DEBUG "status": "success", 05:45:46 DEBUG "data": { 05:45:46 DEBUG "data": { 05:45:46 DEBUG "resultType": "vector", 05:45:46 DEBUG "resultType": "vector", 05:45:46 DEBUG "result": [ 05:45:46 DEBUG "result": [ 05:45:46 DEBUG { 05:45:46 DEBUG { 05:45:46 DEBUG "metric": { 05:45:46 DEBUG "metric": { 05:45:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:46 DEBUG }, 05:45:46 DEBUG }, 05:45:46 DEBUG "value": [ 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG [...] 05:45:46 DEBUG [...] 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:46 DEBUG "pod": "ds-idrepo-0" 05:45:46 DEBUG }, 05:45:46 DEBUG }, 05:45:46 DEBUG "value": [ 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 1713588346, 05:45:46 DEBUG 1713588346, 05:45:46 DEBUG "2677.3333333333335" 05:45:46 DEBUG "0" 05:45:46 DEBUG ] 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:46 DEBUG ] 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588346" 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588346" 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588346" 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588346" 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 DEBUG --- status code --- 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 DEBUG --- status code --- 05:45:46 DEBUG --- http response --- 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 DEBUG --- http response --- 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:45:46 DEBUG --- status code --- 05:45:46 DEBUG ----- output ----- 05:45:46 DEBUG { 05:45:46 DEBUG "status": "success", 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 DEBUG ----- output ----- 05:45:46 DEBUG "data": { 05:45:46 DEBUG --- http response --- 05:45:46 DEBUG { 05:45:46 DEBUG "resultType": "vector", 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:46 DEBUG "status": "success", 05:45:46 DEBUG "result": [ 05:45:46 DEBUG --- status code --- 05:45:46 DEBUG ----- output ----- 05:45:46 DEBUG "data": { 05:45:46 DEBUG { 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 DEBUG { 05:45:46 DEBUG "resultType": "vector", 05:45:46 DEBUG "metric": { 05:45:46 DEBUG --- http response --- 05:45:46 DEBUG "status": "success", 05:45:46 DEBUG "result": [ 05:45:46 DEBUG { 05:45:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:46 DEBUG "data": { 05:45:46 DEBUG "metric": { 05:45:46 DEBUG "instance": "10.142.0.107:9100" 05:45:46 DEBUG ----- output ----- 05:45:46 DEBUG "resultType": "vector", 05:45:46 DEBUG "result": [ 05:45:46 DEBUG }, 05:45:46 DEBUG { 05:45:46 DEBUG { 05:45:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 05:45:46 DEBUG "metric": { 05:45:46 DEBUG }, 05:45:46 DEBUG 05:45:46 DEBUG "status": "success", 05:45:46 DEBUG "instance": "10.142.0.107:9100" 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 05:45:46 DEBUG "data": { 05:45:46 DEBUG }, 05:45:46 DEBUG 05:45:46 DEBUG [...] 05:45:46 DEBUG "resultType": "vector", 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG "result": [ 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG { 05:45:46 DEBUG 05:45:46 DEBUG [...] 05:45:46 DEBUG 05:45:46 DEBUG "metric": { 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG "instance": "10.142.0.43:9100" 05:45:46 DEBUG "pod": "ds-idrepo-0" 05:45:46 DEBUG [...] 05:45:46 DEBUG 05:45:46 DEBUG }, 05:45:46 DEBUG }, 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG "value": [ 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 05:45:46 DEBUG "pod": "ds-idrepo-0" 05:45:46 DEBUG 1713588346, 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG }, 05:45:46 DEBUG "80799123" 05:45:46 DEBUG 05:45:46 DEBUG "instance": "10.142.0.43:9100" 05:45:46 DEBUG "value": [ 05:45:46 DEBUG ] 05:45:46 DEBUG 05:45:46 DEBUG }, 05:45:46 DEBUG 1713588346, 05:45:46 DEBUG } 05:45:46 DEBUG [...] 05:45:46 DEBUG "value": [ 05:45:46 DEBUG "0" 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG 1713588346, 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG "0.11499999999999583" 05:45:46 DEBUG } 05:45:46 DEBUG 05:45:46 DEBUG ] 05:45:46 DEBUG ] 05:45:46 DEBUG 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:46 DEBUG 05:45:46 DEBUG "pod": "ds-idrepo-2" 05:45:46 DEBUG } 05:45:46 DEBUG }, 05:45:46 DEBUG ] 05:45:46 DEBUG "value": [ 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:46 DEBUG 1713588346, 05:45:46 DEBUG "0" 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588346" 05:45:46 INFO [http_cmd]: http status code OK 05:45:46 DEBUG --- status code --- 05:45:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:46 DEBUG --- http response --- 05:45:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:46 DEBUG ----- output ----- 05:45:46 DEBUG { 05:45:46 DEBUG "status": "success", 05:45:46 DEBUG "data": { 05:45:46 DEBUG "resultType": "vector", 05:45:46 DEBUG "result": [ 05:45:46 DEBUG { 05:45:46 DEBUG "metric": { 05:45:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:46 DEBUG }, 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG [...] 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG 05:45:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:46 DEBUG }, 05:45:46 DEBUG "value": [ 05:45:46 DEBUG 1713588346, 05:45:46 DEBUG "0" 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG ] 05:45:46 DEBUG } 05:45:46 DEBUG } 05:45:47 INFO 05:45:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:45:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:47 INFO 05:45:47 INFO [loop_until]: kubectl --namespace=xlou top pods 05:45:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588347" 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- status code --- 05:45:47 INFO [loop_until]: OK (rc = 0) 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG --- status code --- 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- stdout --- 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG --- http response --- 05:45:47 INFO [http_cmd]: http status code OK 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 18085m 8653Mi am-767f6b67b5-qcqz2 17216m 8711Mi am-767f6b67b5-vbltz 18523m 8647Mi ds-cts-0 18534m 10630Mi ds-cts-1 17107m 10719Mi ds-cts-2 18847m 10654Mi ds-idrepo-0 7639m 3720Mi ds-idrepo-1 7609m 3687Mi ds-idrepo-2 8224m 3545Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7894m 7377Mi 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:47 DEBUG --- status code --- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:47 DEBUG --- stderr --- 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG { 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG { 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:47 DEBUG { 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG { 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:47 INFO [loop_until]: OK (rc = 0) 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG { 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG --- stdout --- 05:45:47 DEBUG { 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG { 05:45:47 DEBUG "result": [ 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG "data": { 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG { 05:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:45:47 DEBUG "data": { 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6196Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7906m 26% 7397Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 16871m 56% 11361Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3395Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18812m 63% 13761Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5222Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17579m 58% 11592Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18765m 62% 3012Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19388m 64% 10869Mi 9% gke-xlou-cdm-ds-0f601751-j488 8207m 27% 6839Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17487m 58% 15700Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8012m 26% 7573Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18940m 63% 13261Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8185m 51% 11195Mi 19% 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG "data": { 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "data": { 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG --- http response --- 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG { 05:45:47 DEBUG --- stderr --- 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:45:47 DEBUG { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG { 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:47 DEBUG }, 05:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "data": { 05:45:47 DEBUG "result": [ 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG "result": [ 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG { 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG 05:45:47 DEBUG { 05:45:47 DEBUG "data": { 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG { 05:45:47 DEBUG { 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG ----- output ----- 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "data": { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "instance": "10.142.0.107:9100" 05:45:47 DEBUG "result": [ 05:45:47 DEBUG 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG "pod": "ds-idrepo-0" 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "data": { 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG }, 05:45:47 DEBUG { 05:45:47 DEBUG 05:45:47 DEBUG "pod": "ds-idrepo-0" 05:45:47 DEBUG "result": [ 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "instance": "10.142.0.107:9100" 05:45:47 DEBUG "data": { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "pod": "ds-cts-0" 05:45:47 DEBUG { 05:45:47 DEBUG "status": "success", 05:45:47 DEBUG }, 05:45:47 DEBUG "result": [ 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG "result": [ 05:45:47 DEBUG { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG [...] 05:45:47 DEBUG }, 05:45:47 DEBUG { 05:45:47 DEBUG }, 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG }, 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "pod": "ds-idrepo-0" 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG "instance": "10.142.0.107:9100" 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "result": [ 05:45:47 DEBUG "result": [ 05:45:47 DEBUG 05:45:47 DEBUG "data": { 05:45:47 DEBUG 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "value": [ 05:45:47 DEBUG { 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "pod": "ds-cts-0" 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG { 05:45:47 DEBUG { 05:45:47 DEBUG "pod": "ds-cts-2" 05:45:47 DEBUG "resultType": "vector", 05:45:47 DEBUG 05:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:47 DEBUG 05:45:47 DEBUG "metric": { 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG "metric": { 05:45:47 DEBUG "metric": { 05:45:47 DEBUG }, 05:45:47 DEBUG "result": [ 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:45:47 DEBUG "pod": "ds-cts-0" 05:45:47 DEBUG "value": [ 05:45:47 DEBUG { 05:45:47 DEBUG [...] 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG "0" 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG "value": [ 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG ] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "metric": { 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG [...] 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG 05:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "2665.633333333333" 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:47 DEBUG [...] 05:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:47 DEBUG "instance": "10.142.0.43:9100" 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG } 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG "0" 05:45:47 DEBUG "instance": "10.142.0.43:9100" 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG } 05:45:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:45:47 DEBUG 05:45:47 DEBUG }, 05:45:47 DEBUG "pod": "ds-cts-2" 05:45:47 DEBUG 05:45:47 DEBUG ] 05:45:47 DEBUG 05:45:47 DEBUG ] 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "pod": "ds-idrepo-2" 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "instance": "10.142.0.43:9100" 05:45:47 DEBUG }, 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG [...] 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG } 05:45:47 DEBUG "pod": "ds-idrepo-2" 05:45:47 DEBUG } 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG }, 05:45:47 DEBUG 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "value": [ 05:45:47 DEBUG [...] 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "value": [ 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 05:45:47 DEBUG ] 05:45:47 DEBUG }, 05:45:47 DEBUG ] 05:45:47 DEBUG "80793474.23333333" 05:45:47 DEBUG "value": [ 05:45:47 DEBUG "pod": "ds-cts-2" 05:45:47 DEBUG "2664.733333333333" 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG }, 05:45:47 DEBUG ] 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG "0.092" 05:45:47 DEBUG "0.29986666666667927" 05:45:47 DEBUG 05:45:47 DEBUG 05:45:47 DEBUG "7832879104" 05:45:47 DEBUG 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG "0" 05:45:47 DEBUG } 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG "0" 05:45:47 DEBUG "value": [ 05:45:47 DEBUG } 05:45:47 DEBUG "3076317641728" 05:45:47 DEBUG ] 05:45:47 DEBUG ] 05:45:47 DEBUG 05:45:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG "479917.5" 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG "8.045713655876964" 05:45:47 DEBUG ] 05:45:47 DEBUG ] 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG ] 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG "pod": "ds-cts-2" 05:45:47 DEBUG }, 05:45:47 DEBUG ] 05:45:47 DEBUG 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG "0" 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG "0" 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG ] 05:45:47 DEBUG }, 05:45:47 DEBUG "value": [ 05:45:47 DEBUG } 05:45:47 DEBUG [...] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG ] 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG "value": [ 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG } 05:45:47 DEBUG 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG "0" 05:45:47 DEBUG 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG "0.0007333333333333332" 05:45:47 DEBUG ] 05:45:47 DEBUG 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG ] 05:45:47 DEBUG }, 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG "value": [ 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG 1713588347, 05:45:47 DEBUG } 05:45:47 DEBUG } 05:45:47 DEBUG "2664.7" 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG ] 05:45:47 DEBUG } 05:45:47 DEBUG } 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588406" 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588406" 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588406" 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588406" 05:46:46 ERROR [http_cmd]: ERROR 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 ERROR --- status code --- 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 DEBUG --- status code --- 05:46:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:46:46 DEBUG --- status code --- 05:46:46 DEBUG --- status code --- 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 ERROR --- http response --- 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 DEBUG --- http response --- 05:46:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:46:46 DEBUG --- http response --- 05:46:46 DEBUG --- http response --- 05:46:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:46:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:46:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:46:46 DEBUG ----- output ----- 05:46:46 DEBUG ----- output ----- 05:46:46 DEBUG { 05:46:46 DEBUG { 05:46:46 DEBUG "status": "success", 05:46:46 DEBUG "status": "success", 05:46:46 DEBUG "data": { 05:46:46 DEBUG "data": { 05:46:46 DEBUG "resultType": "vector", 05:46:46 DEBUG "resultType": "vector", 05:46:46 DEBUG "result": [ 05:46:46 DEBUG "result": [ 05:46:46 DEBUG { 05:46:46 DEBUG { 05:46:46 DEBUG "metric": { 05:46:46 DEBUG "metric": { 05:46:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:46 DEBUG }, 05:46:46 DEBUG }, 05:46:46 DEBUG "value": [ 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG [...] 05:46:46 DEBUG [...] 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:46 DEBUG }, 05:46:46 DEBUG }, 05:46:46 DEBUG "value": [ 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 1713588406, 05:46:46 DEBUG "2669.9" 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG 1713588406, 05:46:46 DEBUG } 05:46:46 DEBUG "15319.127537871645" 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588406" 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588406" 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588406" 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588406" 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 DEBUG --- status code --- 05:46:46 DEBUG --- status code --- 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 DEBUG --- http response --- 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 DEBUG --- http response --- 05:46:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:46:46 DEBUG ----- output ----- 05:46:46 DEBUG --- status code --- 05:46:46 DEBUG ----- output ----- 05:46:46 DEBUG { 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 DEBUG { 05:46:46 DEBUG "status": "success", 05:46:46 DEBUG --- http response --- 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 DEBUG "status": "success", 05:46:46 DEBUG "data": { 05:46:46 DEBUG --- status code --- 05:46:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:46 DEBUG "data": { 05:46:46 DEBUG "resultType": "vector", 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 DEBUG ----- output ----- 05:46:46 DEBUG "resultType": "vector", 05:46:46 DEBUG "result": [ 05:46:46 DEBUG --- http response --- 05:46:46 DEBUG { 05:46:46 DEBUG "result": [ 05:46:46 DEBUG { 05:46:46 DEBUG "status": "success", 05:46:46 DEBUG { 05:46:46 DEBUG "metric": { 05:46:46 DEBUG "metric": { 05:46:46 DEBUG "data": { 05:46:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:46 DEBUG "instance": "10.142.0.107:9100" 05:46:46 DEBUG "resultType": "vector", 05:46:46 DEBUG }, 05:46:46 DEBUG ----- output ----- 05:46:46 DEBUG }, 05:46:46 DEBUG "result": [ 05:46:46 DEBUG "value": [ 05:46:46 DEBUG { 05:46:46 DEBUG "value": [ 05:46:46 DEBUG { 05:46:46 DEBUG 05:46:46 DEBUG "status": "success", 05:46:46 DEBUG 05:46:46 DEBUG "metric": { 05:46:46 DEBUG 05:46:46 DEBUG "data": { 05:46:46 DEBUG 05:46:46 DEBUG "instance": "10.142.0.107:9100" 05:46:46 DEBUG 05:46:46 DEBUG "resultType": "vector", 05:46:46 DEBUG 05:46:46 DEBUG }, 05:46:46 DEBUG [...] 05:46:46 DEBUG "result": [ 05:46:46 DEBUG [...] 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 05:46:46 DEBUG { 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG "metric": { 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG "pod": "ds-idrepo-0" 05:46:46 DEBUG 05:46:46 DEBUG "instance": "10.142.0.43:9100" 05:46:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:46 DEBUG }, 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 05:46:46 DEBUG }, 05:46:46 DEBUG }, 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG "value": [ 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 05:46:46 DEBUG [...] 05:46:46 DEBUG 1713588406, 05:46:46 DEBUG 1713588406, 05:46:46 DEBUG [...] 05:46:46 DEBUG 05:46:46 DEBUG "18.69701244578562" 05:46:46 DEBUG "80507007.5" 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG ] 05:46:46 DEBUG ] 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 DEBUG 05:46:46 DEBUG "instance": "10.142.0.43:9100" 05:46:46 DEBUG ] 05:46:46 DEBUG ] 05:46:46 DEBUG "pod": "ds-idrepo-2" 05:46:46 DEBUG }, 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 DEBUG }, 05:46:46 DEBUG "value": [ 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 1713588406, 05:46:46 DEBUG 1713588406, 05:46:46 DEBUG "0.06400000000000432" 05:46:46 DEBUG "0" 05:46:46 DEBUG ] 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 DEBUG ] 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588406" 05:46:46 INFO [http_cmd]: http status code OK 05:46:46 DEBUG --- status code --- 05:46:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:46 DEBUG --- http response --- 05:46:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:46 DEBUG ----- output ----- 05:46:46 DEBUG { 05:46:46 DEBUG "status": "success", 05:46:46 DEBUG "data": { 05:46:46 DEBUG "resultType": "vector", 05:46:46 DEBUG "result": [ 05:46:46 DEBUG { 05:46:46 DEBUG "metric": { 05:46:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:46 DEBUG }, 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG [...] 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG 05:46:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:46 DEBUG }, 05:46:46 DEBUG "value": [ 05:46:46 DEBUG 1713588406, 05:46:46 DEBUG "0" 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG ] 05:46:46 DEBUG } 05:46:46 DEBUG } 05:46:47 INFO 05:46:47 INFO [loop_until]: kubectl --namespace=xlou top pods 05:46:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:47 INFO 05:46:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:46:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588407" 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588407" 05:46:47 DEBUG ----- output ----- 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588407" 05:46:47 INFO [loop_until]: OK (rc = 0) 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG { 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [loop_until]: OK (rc = 0) 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- stdout --- 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- stdout --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17190m 8646Mi am-767f6b67b5-qcqz2 17459m 8709Mi am-767f6b67b5-vbltz 18458m 8649Mi ds-cts-0 18596m 10608Mi ds-cts-1 17185m 10715Mi ds-cts-2 19070m 10665Mi ds-idrepo-0 7704m 3721Mi ds-idrepo-1 7713m 3703Mi ds-idrepo-2 7716m 3553Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 15m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7962m 7376Mi 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG "data": { 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 INFO [http_cmd]: http status code OK 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6205Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7875m 26% 7397Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17415m 58% 11370Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3393Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18618m 62% 13753Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5220Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17561m 58% 11597Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18727m 62% 3004Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19318m 64% 10867Mi 9% gke-xlou-cdm-ds-0f601751-j488 8224m 27% 6865Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16914m 56% 15707Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7999m 26% 7594Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19387m 64% 13256Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8179m 51% 11191Mi 19% 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- stderr --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- status code --- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- stderr --- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:47 DEBUG "result": [ 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:47 DEBUG { 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:46:47 DEBUG --- http response --- 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:47 DEBUG { 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG { 05:46:47 DEBUG "data": { 05:46:47 DEBUG { 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:46:47 DEBUG { 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "pod": "ds-cts-0" 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG { 05:46:47 DEBUG ----- output ----- 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG "data": { 05:46:47 DEBUG "data": { 05:46:47 DEBUG { 05:46:47 DEBUG "data": { 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG }, 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "status": "success", 05:46:47 DEBUG "data": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "data": { 05:46:47 DEBUG { 05:46:47 DEBUG "value": [ 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG "result": [ 05:46:47 DEBUG { 05:46:47 DEBUG "data": { 05:46:47 DEBUG "data": { 05:46:47 DEBUG "pod": "ds-cts-0" 05:46:47 DEBUG "data": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG }, 05:46:47 DEBUG "data": { 05:46:47 DEBUG "data": { 05:46:47 DEBUG "result": [ 05:46:47 DEBUG { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "result": [ 05:46:47 DEBUG 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG }, 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "pod": "ds-idrepo-0" 05:46:47 DEBUG { 05:46:47 DEBUG 05:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:47 DEBUG "instance": "10.142.0.107:9100" 05:46:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "resultType": "vector", 05:46:47 DEBUG "result": [ 05:46:47 DEBUG 05:46:47 DEBUG "result": [ 05:46:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:47 DEBUG "result": [ 05:46:47 DEBUG }, 05:46:47 DEBUG "metric": { 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG 05:46:47 DEBUG { 05:46:47 DEBUG { 05:46:47 DEBUG 05:46:47 DEBUG "instance": "10.142.0.107:9100" 05:46:47 DEBUG { 05:46:47 DEBUG }, 05:46:47 DEBUG { 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:47 DEBUG [...] 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "metric": { 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG "metric": { 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "metric": { 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:46:47 DEBUG "pod": "ds-idrepo-0" 05:46:47 DEBUG "pod": "ds-cts-0" 05:46:47 DEBUG "instance": "10.142.0.107:9100" 05:46:47 DEBUG [...] 05:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:47 DEBUG }, 05:46:47 DEBUG [...] 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:47 DEBUG 05:46:47 DEBUG "pod": "ds-idrepo-0" 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG "pod": "ds-cts-2" 05:46:47 DEBUG [...] 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "pod": "ds-cts-2" 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG }, 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "instance": "10.142.0.43:9100" 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:47 DEBUG "pod": "ds-cts-2" 05:46:47 DEBUG 05:46:47 DEBUG "0" 05:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:47 DEBUG [...] 05:46:47 DEBUG }, 05:46:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG [...] 05:46:47 DEBUG }, 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG ] 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "0" 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "8.044870437899602" 05:46:47 DEBUG "instance": "10.142.0.43:9100" 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:47 DEBUG } 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG "value": [ 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG "pod": "ds-cts-2" 05:46:47 DEBUG ] 05:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:47 DEBUG 05:46:47 DEBUG ] 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG }, 05:46:47 DEBUG ] 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG 05:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG "pod": "ds-idrepo-2" 05:46:47 DEBUG 05:46:47 DEBUG }, 05:46:47 DEBUG } 05:46:47 DEBUG }, 05:46:47 DEBUG 05:46:47 DEBUG } 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:47 DEBUG 05:46:47 DEBUG "1029.8573134703627" 05:46:47 DEBUG "0" 05:46:47 DEBUG "value": [ 05:46:47 DEBUG } 05:46:47 DEBUG "2661.6666666666665" 05:46:47 DEBUG "80502218.43333334" 05:46:47 DEBUG }, 05:46:47 DEBUG "7831642112" 05:46:47 DEBUG }, 05:46:47 DEBUG "instance": "10.142.0.43:9100" 05:46:47 DEBUG "value": [ 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:46:47 DEBUG ] 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG }, 05:46:47 DEBUG "pod": "ds-idrepo-2" 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG "value": [ 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG }, 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG } 05:46:47 DEBUG }, 05:46:47 DEBUG } 05:46:47 DEBUG "3080107746304" 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG }, 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG "2662.9333333333334" 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG "0" 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "value": [ 05:46:47 DEBUG "0.0006344288888888891" 05:46:47 DEBUG "value": [ 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG "value": [ 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG ] 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG } 05:46:47 DEBUG "479858.5" 05:46:47 DEBUG "2661.8" 05:46:47 DEBUG 1713588407, 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG "0" 05:46:47 DEBUG "0.17196666666664745" 05:46:47 DEBUG } 05:46:47 DEBUG "0.092" 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG "0" 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG ] 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:46:47 DEBUG } 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588466" 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588466" 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588466" 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588466" 05:47:46 ERROR [http_cmd]: ERROR 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 ERROR --- status code --- 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:47:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:47:46 ERROR --- http response --- 05:47:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:47:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 DEBUG ----- output ----- 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG { 05:47:46 DEBUG "status": "success", 05:47:46 DEBUG "data": { 05:47:46 DEBUG "resultType": "vector", 05:47:46 DEBUG "result": [ 05:47:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:47:46 DEBUG { 05:47:46 DEBUG ----- output ----- 05:47:46 DEBUG "metric": { 05:47:46 DEBUG { 05:47:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:46 DEBUG "status": "success", 05:47:46 DEBUG }, 05:47:46 DEBUG "data": { 05:47:46 DEBUG "value": [ 05:47:46 DEBUG "resultType": "vector", 05:47:46 DEBUG 05:47:46 DEBUG "result": [ 05:47:46 DEBUG 05:47:46 DEBUG { 05:47:46 DEBUG 05:47:46 DEBUG "metric": { 05:47:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:46 DEBUG [...] 05:47:46 DEBUG }, 05:47:46 DEBUG 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:46 DEBUG }, 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 1713588466, 05:47:46 DEBUG "2668.133333333333" 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG } 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG [...] 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:47:46 DEBUG }, 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 1713588466, 05:47:46 DEBUG "55099.348334045" 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG } 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588466" 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588466" 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588466" 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588466" 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG ----- output ----- 05:47:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:47:46 DEBUG ----- output ----- 05:47:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:46 DEBUG { 05:47:46 DEBUG ----- output ----- 05:47:46 DEBUG "status": "success", 05:47:46 DEBUG { 05:47:46 DEBUG "data": { 05:47:46 DEBUG "resultType": "vector", 05:47:46 DEBUG "result": [ 05:47:46 DEBUG { 05:47:46 DEBUG { 05:47:46 DEBUG "metric": { 05:47:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:46 DEBUG "status": "success", 05:47:46 DEBUG "status": "success", 05:47:46 DEBUG }, 05:47:46 DEBUG "data": { 05:47:46 DEBUG "data": { 05:47:46 DEBUG "value": [ 05:47:46 DEBUG "resultType": "vector", 05:47:46 DEBUG "resultType": "vector", 05:47:46 DEBUG 05:47:46 DEBUG "result": [ 05:47:46 DEBUG "result": [ 05:47:46 DEBUG 05:47:46 DEBUG { 05:47:46 DEBUG { 05:47:46 DEBUG 05:47:46 DEBUG "metric": { 05:47:46 DEBUG "metric": { 05:47:46 DEBUG [...] 05:47:46 DEBUG "instance": "10.142.0.107:9100" 05:47:46 DEBUG "instance": "10.142.0.107:9100" 05:47:46 DEBUG 05:47:46 DEBUG }, 05:47:46 DEBUG }, 05:47:46 DEBUG 05:47:46 DEBUG "value": [ 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG }, 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG "value": [ 05:47:46 DEBUG [...] 05:47:46 DEBUG [...] 05:47:46 DEBUG 1713588466, 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG "15.993292093087579" 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG ] 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG } 05:47:46 DEBUG "instance": "10.142.0.43:9100" 05:47:46 DEBUG "instance": "10.142.0.43:9100" 05:47:46 DEBUG ] 05:47:46 DEBUG }, 05:47:46 DEBUG }, 05:47:46 DEBUG } 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 DEBUG "value": [ 05:47:46 DEBUG "value": [ 05:47:46 DEBUG } 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG 1713588466, 05:47:46 DEBUG 1713588466, 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 DEBUG "0.06566666666665999" 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG "80405816.33333334" 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG ] 05:47:46 DEBUG ] 05:47:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:46 DEBUG } 05:47:46 DEBUG } 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG } 05:47:46 DEBUG ----- output ----- 05:47:46 DEBUG } 05:47:46 DEBUG { 05:47:46 DEBUG "status": "success", 05:47:46 DEBUG "data": { 05:47:46 DEBUG "resultType": "vector", 05:47:46 DEBUG "result": [ 05:47:46 DEBUG { 05:47:46 DEBUG "metric": { 05:47:46 DEBUG "pod": "ds-idrepo-0" 05:47:46 DEBUG }, 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG [...] 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG "pod": "ds-idrepo-2" 05:47:46 DEBUG }, 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 1713588466, 05:47:46 DEBUG "0" 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG } 05:47:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588466" 05:47:46 INFO [http_cmd]: http status code OK 05:47:46 DEBUG --- status code --- 05:47:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:46 DEBUG --- http response --- 05:47:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:46 DEBUG ----- output ----- 05:47:46 DEBUG { 05:47:46 DEBUG "status": "success", 05:47:46 DEBUG "data": { 05:47:46 DEBUG "resultType": "vector", 05:47:46 DEBUG "result": [ 05:47:46 DEBUG { 05:47:46 DEBUG "metric": { 05:47:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:46 DEBUG }, 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG [...] 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG 05:47:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:46 DEBUG }, 05:47:46 DEBUG "value": [ 05:47:46 DEBUG 1713588466, 05:47:46 DEBUG "0" 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG ] 05:47:46 DEBUG } 05:47:46 DEBUG } 05:47:47 INFO 05:47:47 INFO [loop_until]: kubectl --namespace=xlou top pods 05:47:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:47 INFO 05:47:47 INFO [loop_until]: kubectl --namespace=xlou top node 05:47:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588468" 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [loop_until]: OK (rc = 0) 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [loop_until]: OK (rc = 0) 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 DEBUG --- status code --- 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG --- status code --- 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 DEBUG --- stdout --- 05:47:48 DEBUG --- status code --- 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- stdout --- 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17014m 8647Mi am-767f6b67b5-qcqz2 17442m 8710Mi am-767f6b67b5-vbltz 18823m 8650Mi ds-cts-0 18683m 10632Mi ds-cts-1 17234m 10715Mi ds-cts-2 18951m 10664Mi ds-idrepo-0 7789m 3721Mi ds-idrepo-1 7642m 3703Mi ds-idrepo-2 7951m 3566Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 16m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8060m 7377Mi 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG --- status code --- 05:47:48 INFO [http_cmd]: http status code OK 05:47:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 6189Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7866m 26% 7390Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17294m 57% 11367Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3392Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17869m 59% 13753Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17597m 58% 11600Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18541m 62% 2984Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19405m 64% 10856Mi 9% gke-xlou-cdm-ds-0f601751-j488 8225m 27% 6886Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16912m 56% 15712Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8009m 26% 7593Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18852m 63% 13254Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8193m 51% 11195Mi 19% 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- status code --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:48 DEBUG --- stderr --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG { 05:47:48 DEBUG --- stderr --- 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG { 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:48 DEBUG { 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG --- http response --- 05:47:48 DEBUG { 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG { 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:47:48 DEBUG "data": { 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG { 05:47:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG "data": { 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "data": { 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG { 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "data": { 05:47:48 DEBUG { 05:47:48 DEBUG "data": { 05:47:48 DEBUG { 05:47:48 DEBUG { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG { 05:47:48 DEBUG "data": { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "data": { 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "data": { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG ----- output ----- 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "data": { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG { 05:47:48 DEBUG "data": { 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "data": { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG { 05:47:48 DEBUG "data": { 05:47:48 DEBUG "data": { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "data": { 05:47:48 DEBUG "data": { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "data": { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "data": { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG { 05:47:48 DEBUG "status": "success", 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG { 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "metric": { 05:47:48 DEBUG { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "result": [ 05:47:48 DEBUG { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "data": { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "result": [ 05:47:48 DEBUG }, 05:47:48 DEBUG { 05:47:48 DEBUG { 05:47:48 DEBUG { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "pod": "ds-cts-0" 05:47:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:47:48 DEBUG "result": [ 05:47:48 DEBUG { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG { 05:47:48 DEBUG "pod": "ds-idrepo-0" 05:47:48 DEBUG }, 05:47:48 DEBUG { 05:47:48 DEBUG { 05:47:48 DEBUG "instance": "10.142.0.107:9100" 05:47:48 DEBUG { 05:47:48 DEBUG { 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:47:48 DEBUG }, 05:47:48 DEBUG }, 05:47:48 DEBUG { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "resultType": "vector", 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG }, 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "metric": { 05:47:48 DEBUG 05:47:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:48 DEBUG "pod": "ds-cts-0" 05:47:48 DEBUG }, 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "pod": "ds-idrepo-0" 05:47:48 DEBUG }, 05:47:48 DEBUG "instance": "10.142.0.107:9100" 05:47:48 DEBUG 05:47:48 DEBUG "result": [ 05:47:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:48 DEBUG "instance": "10.142.0.107:9100" 05:47:48 DEBUG }, 05:47:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:47:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:47:48 DEBUG 05:47:48 DEBUG }, 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "pod": "ds-cts-0" 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG }, 05:47:48 DEBUG 05:47:48 DEBUG { 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG }, 05:47:48 DEBUG }, 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG "metric": { 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG "pod": "ds-idrepo-0" 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG }, 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG [...] 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "pod": "ds-cts-2" 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "pod": "ds-cts-2" 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:48 DEBUG }, 05:47:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG }, 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG }, 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG 05:47:48 DEBUG "instance": "10.142.0.43:9100" 05:47:48 DEBUG [...] 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG }, 05:47:48 DEBUG 05:47:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG [...] 05:47:48 DEBUG 05:47:48 DEBUG "pod": "ds-cts-0" 05:47:48 DEBUG "pod": "ds-cts-0" 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:48 DEBUG "pod": "ds-cts-2" 05:47:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:47:48 DEBUG }, 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "pod": "ds-idrepo-2" 05:47:48 DEBUG }, 05:47:48 DEBUG "instance": "10.142.0.43:9100" 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG 05:47:48 DEBUG "instance": "10.142.0.43:9100" 05:47:48 DEBUG }, 05:47:48 DEBUG }, 05:47:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG }, 05:47:48 DEBUG }, 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG "3" 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG }, 05:47:48 DEBUG "0" 05:47:48 DEBUG 05:47:48 DEBUG }, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "value": [ 05:47:48 DEBUG }, 05:47:48 DEBUG "2659" 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "value": [ 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG 05:47:48 DEBUG "479809.5" 05:47:48 DEBUG ] 05:47:48 DEBUG "7834247168" 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG ] 05:47:48 DEBUG 05:47:48 DEBUG "value": [ 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG "value": [ 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG "0.17079999999998563" 05:47:48 DEBUG "pod": "ds-cts-2" 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG "2658.9333333333334" 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG "pod": "ds-idrepo-2" 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG "62470610.1122039" 05:47:48 DEBUG "55450727.690815896" 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG ] 05:47:48 DEBUG "2658.8" 05:47:48 DEBUG "8.040276685685793" 05:47:48 DEBUG ] 05:47:48 DEBUG }, 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG "0" 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG }, 05:47:48 DEBUG "3083491419136" 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG "0.092" 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG "value": [ 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG "80400764.73333333" 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG "value": [ 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG "0" 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG 1713588468, 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG "0.0003333333333333334" 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG "0" 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG ] 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:47:48 DEBUG } 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588526" 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588526" 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588526" 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588526" 05:48:46 ERROR [http_cmd]: ERROR 05:48:46 ERROR --- status code --- 05:48:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:48:46 ERROR --- http response --- 05:48:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:48:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 DEBUG --- status code --- 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 DEBUG --- status code --- 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 DEBUG --- status code --- 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:48:46 DEBUG ----- output ----- 05:48:46 DEBUG ----- output ----- 05:48:46 DEBUG { 05:48:46 DEBUG { 05:48:46 DEBUG "status": "success", 05:48:46 DEBUG "status": "success", 05:48:46 DEBUG "data": { 05:48:46 DEBUG "data": { 05:48:46 DEBUG "resultType": "vector", 05:48:46 DEBUG "resultType": "vector", 05:48:46 DEBUG "result": [ 05:48:46 DEBUG "result": [ 05:48:46 DEBUG { 05:48:46 DEBUG { 05:48:46 DEBUG "metric": { 05:48:46 DEBUG "metric": { 05:48:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:46 DEBUG }, 05:48:46 DEBUG }, 05:48:46 DEBUG "value": [ 05:48:46 DEBUG "value": [ 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG [...] 05:48:46 DEBUG [...] 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:48:46 DEBUG }, 05:48:46 DEBUG }, 05:48:46 DEBUG "value": [ 05:48:46 DEBUG "value": [ 05:48:46 DEBUG 1713588526, 05:48:46 DEBUG 1713588526, 05:48:46 DEBUG "2658.9" 05:48:46 DEBUG "267657996.32575515" 05:48:46 DEBUG ] 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG ] 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588526" 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588526" 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588526" 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588526" 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 DEBUG --- status code --- 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:48:46 DEBUG ----- output ----- 05:48:46 DEBUG { 05:48:46 DEBUG "status": "success", 05:48:46 DEBUG "data": { 05:48:46 DEBUG "resultType": "vector", 05:48:46 DEBUG "result": [ 05:48:46 DEBUG { 05:48:46 DEBUG "metric": { 05:48:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:46 DEBUG }, 05:48:46 DEBUG "value": [ 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG [...] 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:48:46 DEBUG }, 05:48:46 DEBUG "value": [ 05:48:46 DEBUG 1713588526, 05:48:46 DEBUG "15.998477547793012" 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 DEBUG --- status code --- 05:48:46 DEBUG --- status code --- 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG --- status code --- 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:46 DEBUG ----- output ----- 05:48:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:46 DEBUG ----- output ----- 05:48:46 DEBUG { 05:48:46 DEBUG ----- output ----- 05:48:46 DEBUG "status": "success", 05:48:46 DEBUG { 05:48:46 DEBUG { 05:48:46 DEBUG "data": { 05:48:46 DEBUG "status": "success", 05:48:46 DEBUG "status": "success", 05:48:46 DEBUG "resultType": "vector", 05:48:46 DEBUG "data": { 05:48:46 DEBUG "data": { 05:48:46 DEBUG "result": [ 05:48:46 DEBUG "resultType": "vector", 05:48:46 DEBUG "resultType": "vector", 05:48:46 DEBUG { 05:48:46 DEBUG "result": [ 05:48:46 DEBUG "result": [ 05:48:46 DEBUG "metric": { 05:48:46 DEBUG { 05:48:46 DEBUG { 05:48:46 DEBUG "instance": "10.142.0.107:9100" 05:48:46 DEBUG "metric": { 05:48:46 DEBUG "metric": { 05:48:46 DEBUG }, 05:48:46 DEBUG "pod": "ds-idrepo-0" 05:48:46 DEBUG "instance": "10.142.0.107:9100" 05:48:46 DEBUG "value": [ 05:48:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588526" 05:48:46 DEBUG }, 05:48:46 DEBUG }, 05:48:46 DEBUG 05:48:46 DEBUG "value": [ 05:48:46 DEBUG "value": [ 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG [...] 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG [...] 05:48:46 DEBUG [...] 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG "instance": "10.142.0.43:9100" 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG }, 05:48:46 DEBUG "pod": "ds-idrepo-2" 05:48:46 DEBUG "instance": "10.142.0.43:9100" 05:48:46 DEBUG "value": [ 05:48:46 DEBUG }, 05:48:46 DEBUG }, 05:48:46 DEBUG 1713588526, 05:48:46 DEBUG "value": [ 05:48:46 DEBUG "value": [ 05:48:46 DEBUG "0.08366666666666259" 05:48:46 DEBUG 1713588526, 05:48:46 DEBUG 1713588526, 05:48:46 DEBUG ] 05:48:46 DEBUG "0" 05:48:46 DEBUG "80466111.33333333" 05:48:46 DEBUG } 05:48:46 DEBUG ] 05:48:46 DEBUG ] 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG ] 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:46 INFO [http_cmd]: http status code OK 05:48:46 DEBUG --- status code --- 05:48:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:46 DEBUG --- http response --- 05:48:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:46 DEBUG ----- output ----- 05:48:46 DEBUG { 05:48:46 DEBUG "status": "success", 05:48:46 DEBUG "data": { 05:48:46 DEBUG "resultType": "vector", 05:48:46 DEBUG "result": [ 05:48:46 DEBUG { 05:48:46 DEBUG "metric": { 05:48:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:46 DEBUG }, 05:48:46 DEBUG "value": [ 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG [...] 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG 05:48:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:46 DEBUG }, 05:48:46 DEBUG "value": [ 05:48:46 DEBUG 1713588526, 05:48:46 DEBUG "0" 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG ] 05:48:46 DEBUG } 05:48:46 DEBUG } 05:48:48 INFO 05:48:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:48:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:48 INFO 05:48:48 INFO [loop_until]: kubectl --namespace=xlou top pods 05:48:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588528" 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 DEBUG --- status code --- 05:48:48 INFO [loop_until]: OK (rc = 0) 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [loop_until]: OK (rc = 0) 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 DEBUG --- stdout --- 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 DEBUG --- status code --- 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 INFO [http_cmd]: http status code OK 05:48:48 DEBUG --- stdout --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17460m 8652Mi am-767f6b67b5-qcqz2 17198m 8714Mi am-767f6b67b5-vbltz 18924m 8649Mi ds-cts-0 19316m 10618Mi ds-cts-1 17510m 10719Mi ds-cts-2 18728m 10657Mi ds-idrepo-0 7630m 3714Mi ds-idrepo-1 7514m 3703Mi ds-idrepo-2 8063m 3573Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 7m 1177Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-799cdc5bdb-7t5x7 13m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7973m 7377Mi 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6202Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7923m 26% 7384Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17310m 57% 11356Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3404Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18950m 63% 13752Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17549m 58% 11599Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18757m 62% 3000Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19382m 64% 10849Mi 9% gke-xlou-cdm-ds-0f601751-j488 8231m 27% 6891Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17193m 57% 15719Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7973m 26% 7601Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19242m 64% 13252Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8187m 51% 11192Mi 19% 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- stderr --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- status code --- 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- stderr --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 DEBUG 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 DEBUG { 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG --- http response --- 05:48:48 DEBUG { 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG "data": { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG { 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG ----- output ----- 05:48:48 DEBUG "data": { 05:48:48 DEBUG { 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG { 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "data": { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "data": { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "data": { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "data": { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG "status": "success", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "data": { 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "result": [ 05:48:48 DEBUG }, 05:48:48 DEBUG { 05:48:48 DEBUG "result": [ 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "ds-cts-0" 05:48:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:48 DEBUG "resultType": "vector", 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:48:48 DEBUG { 05:48:48 DEBUG { 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "metric": { 05:48:48 DEBUG { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:48 DEBUG "pod": "ds-idrepo-0" 05:48:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:48 DEBUG }, 05:48:48 DEBUG }, 05:48:48 DEBUG "result": [ 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "ds-cts-0" 05:48:48 DEBUG "instance": "10.142.0.107:9100" 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:48 DEBUG }, 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "metric": { 05:48:48 DEBUG 05:48:48 DEBUG "instance": "10.142.0.107:9100" 05:48:48 DEBUG "metric": { 05:48:48 DEBUG "pod": "ds-idrepo-0" 05:48:48 DEBUG }, 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG { 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:48:48 DEBUG "pod": "ds-cts-0" 05:48:48 DEBUG }, 05:48:48 DEBUG }, 05:48:48 DEBUG "instance": "10.142.0.107:9100" 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "metric": { 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "value": [ 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "value": [ 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG "pod": "ds-idrepo-0" 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "pod": "ds-cts-2" 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG [...] 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "pod": "ds-cts-2" 05:48:48 DEBUG 05:48:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG [...] 05:48:48 DEBUG "instance": "10.142.0.43:9100" 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "instance": "10.142.0.43:9100" 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG "2667.4333333333334" 05:48:48 DEBUG 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "0" 05:48:48 DEBUG "pod": "ds-idrepo-2" 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:48:48 DEBUG 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:48:48 DEBUG 05:48:48 DEBUG 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG ] 05:48:48 DEBUG 05:48:48 DEBUG "2660.233333333333" 05:48:48 DEBUG ] 05:48:48 DEBUG }, 05:48:48 DEBUG "0" 05:48:48 DEBUG }, 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG }, 05:48:48 DEBUG "pod": "ds-cts-2" 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:48:48 DEBUG "7833702400" 05:48:48 DEBUG "3087477328896" 05:48:48 DEBUG 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG } 05:48:48 DEBUG "pod": "ds-cts-2" 05:48:48 DEBUG }, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "29.966666666666665" 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG }, 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG }, 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG "instance": "10.142.0.43:9100" 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "0.092" 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "pod": "ds-idrepo-2" 05:48:48 DEBUG "0.1841333333332661" 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG "value": [ 05:48:48 DEBUG "479335.5" 05:48:48 DEBUG "value": [ 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG }, 05:48:48 DEBUG "0" 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG "0" 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG "0" 05:48:48 DEBUG }, 05:48:48 DEBUG ] 05:48:48 DEBUG "8.041723952304265" 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG ] 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG "value": [ 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG "value": [ 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG "0.00023333333333333344" 05:48:48 DEBUG } 05:48:48 DEBUG "2661.5333333333333" 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG "80461920.3" 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG 1713588528, 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG "0" 05:48:48 DEBUG ] 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG } 05:48:48 DEBUG ] 05:48:48 DEBUG } 05:48:48 DEBUG } 05:49:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588586" 05:49:46 ERROR [http_cmd]: ERROR 05:49:46 ERROR --- status code --- 05:49:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:49:46 ERROR --- http response --- 05:49:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:49:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:49:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588586" 05:49:46 INFO [http_cmd]: http status code OK 05:49:46 DEBUG --- status code --- 05:49:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:46 DEBUG --- http response --- 05:49:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:46 DEBUG ----- output ----- 05:49:46 DEBUG { 05:49:46 DEBUG "status": "success", 05:49:46 DEBUG "data": { 05:49:46 DEBUG "resultType": "vector", 05:49:46 DEBUG "result": [ 05:49:46 DEBUG { 05:49:46 DEBUG "metric": { 05:49:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:46 DEBUG }, 05:49:46 DEBUG "value": [ 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG [...] 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:46 DEBUG }, 05:49:46 DEBUG "value": [ 05:49:46 DEBUG 1713588586, 05:49:46 DEBUG "2663.366666666667" 05:49:46 DEBUG ] 05:49:46 DEBUG } 05:49:46 DEBUG ] 05:49:46 DEBUG } 05:49:46 DEBUG } 05:49:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588586" 05:49:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588586" 05:49:46 INFO [http_cmd]: http status code OK 05:49:46 DEBUG --- status code --- 05:49:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:46 DEBUG --- http response --- 05:49:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:49:46 INFO [http_cmd]: http status code OK 05:49:46 DEBUG --- status code --- 05:49:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:46 DEBUG --- http response --- 05:49:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:46 DEBUG ----- output ----- 05:49:46 DEBUG { 05:49:46 DEBUG "status": "success", 05:49:46 DEBUG "data": { 05:49:46 DEBUG "resultType": "vector", 05:49:46 DEBUG "result": [ 05:49:46 DEBUG { 05:49:46 DEBUG "metric": { 05:49:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:46 DEBUG }, 05:49:46 DEBUG "value": [ 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG [...] 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG 05:49:46 DEBUG "pod": "ds-cts-0" 05:49:46 DEBUG }, 05:49:46 DEBUG "value": [ 05:49:46 DEBUG 1713588586, 05:49:46 DEBUG "372136251.08746576" 05:49:46 DEBUG ] 05:49:46 DEBUG } 05:49:46 DEBUG ] 05:49:46 DEBUG } 05:49:46 DEBUG } 05:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588587" 05:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588587" 05:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588587" 05:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588587" 05:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588587" 05:49:47 INFO [http_cmd]: http status code OK 05:49:47 DEBUG --- status code --- 05:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:47 DEBUG --- http response --- 05:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:47 DEBUG ----- output ----- 05:49:47 DEBUG { 05:49:47 DEBUG "status": "success", 05:49:47 DEBUG "data": { 05:49:47 DEBUG "resultType": "vector", 05:49:47 DEBUG "result": [ 05:49:47 DEBUG { 05:49:47 DEBUG "metric": { 05:49:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:47 DEBUG }, 05:49:47 DEBUG "value": [ 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG [...] 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG "pod": "ds-cts-0" 05:49:47 DEBUG }, 05:49:47 DEBUG "value": [ 05:49:47 DEBUG 1713588587, 05:49:47 DEBUG "18.90554714581842" 05:49:47 DEBUG ] 05:49:47 DEBUG } 05:49:47 DEBUG ] 05:49:47 DEBUG } 05:49:47 DEBUG } 05:49:47 INFO [http_cmd]: http status code OK 05:49:47 DEBUG --- status code --- 05:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:47 DEBUG --- http response --- 05:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:47 DEBUG ----- output ----- 05:49:47 DEBUG { 05:49:47 DEBUG "status": "success", 05:49:47 DEBUG "data": { 05:49:47 DEBUG "resultType": "vector", 05:49:47 DEBUG "result": [ 05:49:47 DEBUG { 05:49:47 DEBUG "metric": { 05:49:47 DEBUG "instance": "10.142.0.107:9100" 05:49:47 DEBUG }, 05:49:47 DEBUG "value": [ 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG [...] 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG "instance": "10.142.0.43:9100" 05:49:47 DEBUG }, 05:49:47 DEBUG "value": [ 05:49:47 DEBUG 1713588587, 05:49:47 DEBUG "80797529.8" 05:49:47 DEBUG ] 05:49:47 DEBUG } 05:49:47 DEBUG ] 05:49:47 DEBUG } 05:49:47 DEBUG } 05:49:47 INFO [http_cmd]: http status code OK 05:49:47 DEBUG --- status code --- 05:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:47 DEBUG --- http response --- 05:49:47 INFO [http_cmd]: http status code OK 05:49:47 DEBUG --- status code --- 05:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:47 DEBUG --- http response --- 05:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:47 DEBUG ----- output ----- 05:49:47 DEBUG { 05:49:47 DEBUG "status": "success", 05:49:47 DEBUG "data": { 05:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:47 DEBUG "resultType": "vector", 05:49:47 DEBUG ----- output ----- 05:49:47 DEBUG "result": [ 05:49:47 DEBUG { 05:49:47 INFO [http_cmd]: http status code OK 05:49:47 DEBUG { 05:49:47 DEBUG "status": "success", 05:49:47 DEBUG --- status code --- 05:49:47 DEBUG "metric": { 05:49:47 DEBUG "data": { 05:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:47 DEBUG "instance": "10.142.0.107:9100" 05:49:47 DEBUG "resultType": "vector", 05:49:47 DEBUG --- http response --- 05:49:47 DEBUG }, 05:49:47 DEBUG "result": [ 05:49:47 DEBUG "value": [ 05:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:47 DEBUG { 05:49:47 DEBUG 05:49:47 DEBUG ----- output ----- 05:49:47 DEBUG "metric": { 05:49:47 DEBUG 05:49:47 DEBUG { 05:49:47 DEBUG "pod": "ds-idrepo-0" 05:49:47 DEBUG 05:49:47 DEBUG "status": "success", 05:49:47 DEBUG }, 05:49:47 DEBUG [...] 05:49:47 DEBUG "data": { 05:49:47 DEBUG "value": [ 05:49:47 DEBUG 05:49:47 DEBUG "resultType": "vector", 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG "result": [ 05:49:47 DEBUG 05:49:47 DEBUG 05:49:47 DEBUG { 05:49:47 DEBUG 05:49:47 DEBUG [...] 05:49:47 DEBUG "metric": { 05:49:47 DEBUG 05:49:47 DEBUG "instance": "10.142.0.43:9100" 05:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:47 DEBUG 05:49:47 DEBUG }, 05:49:47 DEBUG }, 05:49:47 DEBUG 05:49:47 DEBUG "value": [ 05:49:47 DEBUG "value": [ 05:49:47 DEBUG "pod": "ds-idrepo-2" 05:49:47 DEBUG 1713588587, 05:49:47 DEBUG 05:49:47 DEBUG }, 05:49:47 DEBUG "0.10233333333333879" 05:49:47 DEBUG 05:49:47 DEBUG "value": [ 05:49:47 DEBUG ] 05:49:47 DEBUG 05:49:47 DEBUG [...] 05:49:47 DEBUG } 05:49:47 DEBUG ] 05:49:47 DEBUG 05:49:47 DEBUG 1713588587, 05:49:47 DEBUG } 05:49:47 DEBUG 05:49:47 DEBUG "0" 05:49:47 DEBUG } 05:49:47 DEBUG 05:49:47 DEBUG ] 05:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:47 DEBUG } 05:49:47 DEBUG }, 05:49:47 DEBUG ] 05:49:47 DEBUG "value": [ 05:49:47 DEBUG } 05:49:47 DEBUG 1713588587, 05:49:47 DEBUG } 05:49:47 DEBUG "0" 05:49:47 DEBUG ] 05:49:47 DEBUG } 05:49:47 DEBUG ] 05:49:47 DEBUG } 05:49:47 DEBUG } 05:49:48 INFO 05:49:48 INFO [loop_until]: kubectl --namespace=xlou top pods 05:49:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:48 INFO 05:49:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:49:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588588" 05:49:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588588" 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [loop_until]: OK (rc = 0) 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [loop_until]: OK (rc = 0) 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- stdout --- 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- status code --- 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- stdout --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 6193Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7910m 26% 7370Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17310m 57% 11373Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3395Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18862m 63% 13764Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5213Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17558m 58% 11600Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18877m 63% 2999Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19333m 64% 10876Mi 9% gke-xlou-cdm-ds-0f601751-j488 8259m 27% 6898Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17291m 57% 15712Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8057m 26% 7577Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19157m 64% 13254Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8186m 51% 11192Mi 19% 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- status code --- 05:49:48 INFO [http_cmd]: http status code OK 05:49:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17201m 8652Mi am-767f6b67b5-qcqz2 17280m 8714Mi am-767f6b67b5-vbltz 18800m 8651Mi ds-cts-0 18895m 10625Mi ds-cts-1 17892m 10717Mi ds-cts-2 19283m 10666Mi ds-idrepo-0 7706m 3714Mi ds-idrepo-1 7427m 3703Mi ds-idrepo-2 7874m 3573Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8206m 7377Mi 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- stderr --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG --- status code --- 05:49:48 DEBUG --- stderr --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:49:48 DEBUG 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG { 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:48 DEBUG { 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:48 DEBUG { 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG { 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:49:48 DEBUG --- http response --- 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG { 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:48 DEBUG { 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:49:48 DEBUG "data": { 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG { 05:49:48 DEBUG "data": { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG { 05:49:48 DEBUG "data": { 05:49:48 DEBUG { 05:49:48 DEBUG "data": { 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "data": { 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG ----- output ----- 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "data": { 05:49:48 DEBUG { 05:49:48 DEBUG "data": { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "data": { 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "data": { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "data": { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG { 05:49:48 DEBUG "status": "success", 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "data": { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "data": { 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "data": { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "metric": { 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "instance": "10.142.0.107:9100" 05:49:48 DEBUG { 05:49:48 DEBUG { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG "result": [ 05:49:48 DEBUG }, 05:49:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:48 DEBUG }, 05:49:48 DEBUG { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "metric": { 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG { 05:49:48 DEBUG "instance": "10.142.0.107:9100" 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG }, 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "pod": "ds-cts-0" 05:49:48 DEBUG { 05:49:48 DEBUG "result": [ 05:49:48 DEBUG { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "pod": "ds-cts-0" 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG { 05:49:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:48 DEBUG "metric": { 05:49:48 DEBUG }, 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "metric": { 05:49:48 DEBUG }, 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "value": [ 05:49:48 DEBUG }, 05:49:48 DEBUG "resultType": "vector", 05:49:48 DEBUG { 05:49:48 DEBUG 05:49:48 DEBUG "metric": { 05:49:48 DEBUG }, 05:49:48 DEBUG "metric": { 05:49:48 DEBUG }, 05:49:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:49:48 DEBUG "pod": "ds-idrepo-0" 05:49:48 DEBUG "value": [ 05:49:48 DEBUG { 05:49:48 DEBUG 05:49:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:48 DEBUG "pod": "ds-idrepo-0" 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "result": [ 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "pod": "ds-cts-0" 05:49:48 DEBUG "instance": "10.142.0.107:9100" 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG }, 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG "metric": { 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG { 05:49:48 DEBUG }, 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "metric": { 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "value": [ 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "pod": "ds-idrepo-0" 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "instance": "10.142.0.43:9100" 05:49:48 DEBUG 05:49:48 DEBUG "pod": "ds-cts-2" 05:49:48 DEBUG 05:49:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "instance": "10.142.0.43:9100" 05:49:48 DEBUG [...] 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG "pod": "ds-cts-2" 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "pod": "ds-cts-2" 05:49:48 DEBUG [...] 05:49:48 DEBUG "value": [ 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "pod": "ds-cts-2" 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG [...] 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG }, 05:49:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:49:48 DEBUG }, 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG "pod": "ds-idrepo-2" 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 05:49:48 DEBUG "2665.9666666666667" 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG 05:49:48 DEBUG "value": [ 05:49:48 DEBUG }, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG }, 05:49:48 DEBUG "0.00010000000000000003" 05:49:48 DEBUG }, 05:49:48 DEBUG "479016" 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG "pod": "ds-cts-0" 05:49:48 DEBUG 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG 05:49:48 DEBUG ] 05:49:48 DEBUG "2665.733333333333" 05:49:48 DEBUG "pod": "ds-cts-0" 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG "value": [ 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG }, 05:49:48 DEBUG "0.24783333333329552" 05:49:48 DEBUG "value": [ 05:49:48 DEBUG ] 05:49:48 DEBUG "value": [ 05:49:48 DEBUG ] 05:49:48 DEBUG "30" 05:49:48 DEBUG }, 05:49:48 DEBUG 05:49:48 DEBUG "0" 05:49:48 DEBUG "instance": "10.142.0.43:9100" 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG }, 05:49:48 DEBUG "2665.8" 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG "0" 05:49:48 DEBUG "3091666516992" 05:49:48 DEBUG "value": [ 05:49:48 DEBUG ] 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG } 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG "value": [ 05:49:48 DEBUG "pod": "ds-idrepo-2" 05:49:48 DEBUG ] 05:49:48 DEBUG }, 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG "value": [ 05:49:48 DEBUG ] 05:49:48 DEBUG "7835410432" 05:49:48 DEBUG ] 05:49:48 DEBUG ] 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG }, 05:49:48 DEBUG } 05:49:48 DEBUG "value": [ 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG "8.026858124377629" 05:49:48 DEBUG } 05:49:48 DEBUG "0.092" 05:49:48 DEBUG "0" 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG "62465439.06846016" 05:49:48 DEBUG ] 05:49:48 DEBUG ] 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG "70541765.37509201" 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG ] 05:49:48 DEBUG ] 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG "value": [ 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG "80794223.5" 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG 1713588588, 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG "0" 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG ] 05:49:48 DEBUG } 05:49:48 DEBUG } 05:49:48 DEBUG } 05:50:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588646" 05:50:46 ERROR [http_cmd]: ERROR 05:50:46 ERROR --- status code --- 05:50:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:50:46 ERROR --- http response --- 05:50:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:50:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:50:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588646" 05:50:46 INFO [http_cmd]: http status code OK 05:50:46 DEBUG --- status code --- 05:50:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:46 DEBUG --- http response --- 05:50:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:46 DEBUG ----- output ----- 05:50:46 DEBUG { 05:50:46 DEBUG "status": "success", 05:50:46 DEBUG "data": { 05:50:46 DEBUG "resultType": "vector", 05:50:46 DEBUG "result": [ 05:50:46 DEBUG { 05:50:46 DEBUG "metric": { 05:50:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:46 DEBUG }, 05:50:46 DEBUG "value": [ 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG [...] 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:46 DEBUG }, 05:50:46 DEBUG "value": [ 05:50:46 DEBUG 1713588646, 05:50:46 DEBUG "2657.1666666666665" 05:50:46 DEBUG ] 05:50:46 DEBUG } 05:50:46 DEBUG ] 05:50:46 DEBUG } 05:50:46 DEBUG } 05:50:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588646" 05:50:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588646" 05:50:46 INFO [http_cmd]: http status code OK 05:50:46 DEBUG --- status code --- 05:50:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:46 DEBUG --- http response --- 05:50:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:50:46 INFO [http_cmd]: http status code OK 05:50:46 DEBUG --- status code --- 05:50:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:46 DEBUG --- http response --- 05:50:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:50:46 DEBUG ----- output ----- 05:50:46 DEBUG { 05:50:46 DEBUG "status": "success", 05:50:46 DEBUG "data": { 05:50:46 DEBUG "resultType": "vector", 05:50:46 DEBUG "result": [ 05:50:46 DEBUG { 05:50:46 DEBUG "metric": { 05:50:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:46 DEBUG }, 05:50:46 DEBUG "value": [ 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG [...] 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG 05:50:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:50:46 DEBUG }, 05:50:46 DEBUG "value": [ 05:50:46 DEBUG 1713588646, 05:50:46 DEBUG "0" 05:50:46 DEBUG ] 05:50:46 DEBUG } 05:50:46 DEBUG ] 05:50:46 DEBUG } 05:50:46 DEBUG } 05:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588647" 05:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588647" 05:50:47 INFO [http_cmd]: http status code OK 05:50:47 DEBUG --- status code --- 05:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:47 DEBUG --- http response --- 05:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:47 DEBUG ----- output ----- 05:50:47 DEBUG { 05:50:47 DEBUG "status": "success", 05:50:47 DEBUG "data": { 05:50:47 DEBUG "resultType": "vector", 05:50:47 DEBUG "result": [ 05:50:47 DEBUG { 05:50:47 DEBUG "metric": { 05:50:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG [...] 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 1713588647, 05:50:47 DEBUG "13.617520989595903" 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG } 05:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588647" 05:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588647" 05:50:47 INFO [http_cmd]: http status code OK 05:50:47 DEBUG --- status code --- 05:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:47 DEBUG --- http response --- 05:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588647" 05:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:47 DEBUG ----- output ----- 05:50:47 DEBUG { 05:50:47 DEBUG "status": "success", 05:50:47 DEBUG "data": { 05:50:47 DEBUG "resultType": "vector", 05:50:47 DEBUG "result": [ 05:50:47 DEBUG { 05:50:47 DEBUG "metric": { 05:50:47 DEBUG "instance": "10.142.0.107:9100" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG [...] 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG "instance": "10.142.0.43:9100" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 1713588647, 05:50:47 DEBUG "80651568.73333333" 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG } 05:50:47 INFO [http_cmd]: http status code OK 05:50:47 DEBUG --- status code --- 05:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:47 DEBUG --- http response --- 05:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:47 DEBUG ----- output ----- 05:50:47 DEBUG { 05:50:47 DEBUG "status": "success", 05:50:47 DEBUG "data": { 05:50:47 DEBUG "resultType": "vector", 05:50:47 DEBUG "result": [ 05:50:47 DEBUG { 05:50:47 DEBUG "metric": { 05:50:47 DEBUG "instance": "10.142.0.107:9100" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG [...] 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG "instance": "10.142.0.43:9100" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 1713588647, 05:50:47 DEBUG "0.06700000000000347" 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG } 05:50:47 INFO [http_cmd]: http status code OK 05:50:47 DEBUG --- status code --- 05:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:47 DEBUG --- http response --- 05:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:47 DEBUG ----- output ----- 05:50:47 DEBUG { 05:50:47 DEBUG "status": "success", 05:50:47 DEBUG "data": { 05:50:47 DEBUG "resultType": "vector", 05:50:47 DEBUG "result": [ 05:50:47 DEBUG { 05:50:47 DEBUG "metric": { 05:50:47 DEBUG "pod": "ds-idrepo-0" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG [...] 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG "pod": "ds-idrepo-2" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 1713588647, 05:50:47 DEBUG "0" 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG } 05:50:47 INFO [http_cmd]: http status code OK 05:50:47 DEBUG --- status code --- 05:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:47 DEBUG --- http response --- 05:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:47 DEBUG ----- output ----- 05:50:47 DEBUG { 05:50:47 DEBUG "status": "success", 05:50:47 DEBUG "data": { 05:50:47 DEBUG "resultType": "vector", 05:50:47 DEBUG "result": [ 05:50:47 DEBUG { 05:50:47 DEBUG "metric": { 05:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG [...] 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG 05:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:47 DEBUG }, 05:50:47 DEBUG "value": [ 05:50:47 DEBUG 1713588647, 05:50:47 DEBUG "0" 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG ] 05:50:47 DEBUG } 05:50:47 DEBUG } 05:50:48 INFO 05:50:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:50:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:48 INFO 05:50:48 INFO [loop_until]: kubectl --namespace=xlou top pods 05:50:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588648" 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [loop_until]: OK (rc = 0) 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 INFO [loop_until]: OK (rc = 0) 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- status code --- 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- status code --- 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- status code --- 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- stdout --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- stdout --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17496m 8650Mi am-767f6b67b5-qcqz2 17497m 8713Mi am-767f6b67b5-vbltz 18878m 8650Mi ds-cts-0 18913m 10622Mi ds-cts-1 17042m 10724Mi ds-cts-2 18917m 10596Mi ds-idrepo-0 7554m 3714Mi ds-idrepo-1 7764m 3701Mi ds-idrepo-2 7879m 3574Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8216m 7378Mi 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6211Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7862m 26% 7375Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17297m 57% 11369Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3378Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18440m 61% 13758Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5195Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17542m 58% 11595Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18730m 62% 2996Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 18791m 62% 10856Mi 9% gke-xlou-cdm-ds-0f601751-j488 8190m 27% 6896Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17323m 58% 15711Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8039m 26% 7595Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18976m 63% 13194Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8184m 51% 11194Mi 19% 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 INFO [http_cmd]: http status code OK 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- stderr --- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG --- status code --- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG { 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:48 DEBUG 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:50:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG --- stderr --- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG { 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG --- http response --- 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:50:48 DEBUG "data": { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG { 05:50:48 DEBUG "data": { 05:50:48 DEBUG "data": { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "data": { 05:50:48 DEBUG "data": { 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG ----- output ----- 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "data": { 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "result": [ 05:50:48 DEBUG { 05:50:48 DEBUG "data": { 05:50:48 DEBUG "result": [ 05:50:48 DEBUG { 05:50:48 DEBUG "data": { 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "data": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG { 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "status": "success", 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "data": { 05:50:48 DEBUG { 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG { 05:50:48 DEBUG "data": { 05:50:48 DEBUG "pod": "ds-cts-0" 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG { 05:50:48 DEBUG { 05:50:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG "metric": { 05:50:48 DEBUG { 05:50:48 DEBUG "pod": "ds-idrepo-0" 05:50:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "resultType": "vector", 05:50:48 DEBUG }, 05:50:48 DEBUG { 05:50:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:48 DEBUG }, 05:50:48 DEBUG { 05:50:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:50:48 DEBUG "instance": "10.142.0.107:9100" 05:50:48 DEBUG "pod": "ds-cts-0" 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "metric": { 05:50:48 DEBUG "instance": "10.142.0.107:9100" 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "instance": "10.142.0.107:9100" 05:50:48 DEBUG "metric": { 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:50:48 DEBUG "result": [ 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "metric": { 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "metric": { 05:50:48 DEBUG }, 05:50:48 DEBUG }, 05:50:48 DEBUG }, 05:50:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:48 DEBUG }, 05:50:48 DEBUG }, 05:50:48 DEBUG { 05:50:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG { 05:50:48 DEBUG 05:50:48 DEBUG "pod": "ds-idrepo-0" 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "value": [ 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "metric": { 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG }, 05:50:48 DEBUG "metric": { 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "pod": "ds-idrepo-0" 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "pod": "ds-cts-0" 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG "instance": "10.142.0.43:9100" 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG "pod": "ds-cts-2" 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "3095665212416" 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "pod": "ds-cts-2" 05:50:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:48 DEBUG [...] 05:50:48 DEBUG ] 05:50:48 DEBUG 05:50:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:50:48 DEBUG }, 05:50:48 DEBUG [...] 05:50:48 DEBUG 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "0" 05:50:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:48 DEBUG "pod": "ds-idrepo-2" 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:50:48 DEBUG ] 05:50:48 DEBUG "instance": "10.142.0.43:9100" 05:50:48 DEBUG }, 05:50:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG }, 05:50:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "instance": "10.142.0.43:9100" 05:50:48 DEBUG } 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 05:50:48 DEBUG 05:50:48 DEBUG "2663.133333333333" 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG }, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG }, 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "2663.6" 05:50:48 DEBUG "pod": "ds-cts-2" 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "value": [ 05:50:48 DEBUG }, 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG }, 05:50:48 DEBUG ] 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "pod": "ds-cts-2" 05:50:48 DEBUG 05:50:48 DEBUG }, 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "0.0001666666666666667" 05:50:48 DEBUG ] 05:50:48 DEBUG }, 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "2663.5" 05:50:48 DEBUG "value": [ 05:50:48 DEBUG } 05:50:48 DEBUG "0" 05:50:48 DEBUG }, 05:50:48 DEBUG 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG "7838068736" 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "479359.5" 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "8.044467342640573" 05:50:48 DEBUG ] 05:50:48 DEBUG ] 05:50:48 DEBUG "value": [ 05:50:48 DEBUG "pod": "ds-idrepo-2" 05:50:48 DEBUG }, 05:50:48 DEBUG ] 05:50:48 DEBUG "0" 05:50:48 DEBUG ] 05:50:48 DEBUG "0.092" 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "0" 05:50:48 DEBUG ] 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "80647319.26666667" 05:50:48 DEBUG ] 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG "value": [ 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG "33.93333333333333" 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG "0" 05:50:48 DEBUG "0" 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG ] 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG 1713588648, 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG "0.16526666666664389" 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG } 05:50:48 DEBUG ] 05:50:48 DEBUG } 05:50:48 DEBUG } 05:51:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588706" 05:51:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588706" 05:51:46 ERROR [http_cmd]: ERROR 05:51:46 ERROR --- status code --- 05:51:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:51:46 ERROR --- http response --- 05:51:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:51:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:51:46 INFO [http_cmd]: http status code OK 05:51:46 DEBUG --- status code --- 05:51:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:46 DEBUG --- http response --- 05:51:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:46 DEBUG ----- output ----- 05:51:46 DEBUG { 05:51:46 DEBUG "status": "success", 05:51:46 DEBUG "data": { 05:51:46 DEBUG "resultType": "vector", 05:51:46 DEBUG "result": [ 05:51:46 DEBUG { 05:51:46 DEBUG "metric": { 05:51:46 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:46 DEBUG }, 05:51:46 DEBUG "value": [ 05:51:46 DEBUG 05:51:46 DEBUG 05:51:46 DEBUG 05:51:46 DEBUG [...] 05:51:46 DEBUG 05:51:46 DEBUG 05:51:46 DEBUG 05:51:46 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:46 DEBUG }, 05:51:46 DEBUG "value": [ 05:51:46 DEBUG 1713588706, 05:51:46 DEBUG "2661.366666666667" 05:51:46 DEBUG ] 05:51:46 DEBUG } 05:51:46 DEBUG ] 05:51:46 DEBUG } 05:51:46 DEBUG } 05:51:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588706" 05:51:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588706" 05:51:46 INFO [http_cmd]: http status code OK 05:51:46 DEBUG --- status code --- 05:51:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:46 DEBUG --- http response --- 05:51:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:51:47 INFO [http_cmd]: http status code OK 05:51:47 DEBUG --- status code --- 05:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:47 DEBUG --- http response --- 05:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:51:47 DEBUG ----- output ----- 05:51:47 DEBUG { 05:51:47 DEBUG "status": "success", 05:51:47 DEBUG "data": { 05:51:47 DEBUG "resultType": "vector", 05:51:47 DEBUG "result": [ 05:51:47 DEBUG { 05:51:47 DEBUG "metric": { 05:51:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG [...] 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG "pod": "ds-cts-2" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 1713588706, 05:51:47 DEBUG "0" 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG } 05:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588707" 05:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588707" 05:51:47 INFO [http_cmd]: http status code OK 05:51:47 DEBUG --- status code --- 05:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:47 DEBUG --- http response --- 05:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:51:47 DEBUG ----- output ----- 05:51:47 DEBUG { 05:51:47 DEBUG "status": "success", 05:51:47 DEBUG "data": { 05:51:47 DEBUG "resultType": "vector", 05:51:47 DEBUG "result": [ 05:51:47 DEBUG { 05:51:47 DEBUG "metric": { 05:51:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588707" 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG [...] 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG "pod": "ds-cts-2" 05:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588707" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588707" 05:51:47 DEBUG 1713588707, 05:51:47 DEBUG "0" 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG } 05:51:47 INFO [http_cmd]: http status code OK 05:51:47 DEBUG --- status code --- 05:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:47 DEBUG --- http response --- 05:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:47 DEBUG ----- output ----- 05:51:47 DEBUG { 05:51:47 DEBUG "status": "success", 05:51:47 DEBUG "data": { 05:51:47 DEBUG "resultType": "vector", 05:51:47 DEBUG "result": [ 05:51:47 DEBUG { 05:51:47 DEBUG "metric": { 05:51:47 DEBUG "instance": "10.142.0.107:9100" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG [...] 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG "instance": "10.142.0.43:9100" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 1713588707, 05:51:47 DEBUG "80505120.5" 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG } 05:51:47 INFO [http_cmd]: http status code OK 05:51:47 DEBUG --- status code --- 05:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:47 DEBUG --- http response --- 05:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:47 DEBUG ----- output ----- 05:51:47 DEBUG { 05:51:47 DEBUG "status": "success", 05:51:47 DEBUG "data": { 05:51:47 DEBUG "resultType": "vector", 05:51:47 DEBUG "result": [ 05:51:47 DEBUG { 05:51:47 DEBUG "metric": { 05:51:47 DEBUG "instance": "10.142.0.107:9100" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG [...] 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG "instance": "10.142.0.43:9100" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 1713588707, 05:51:47 DEBUG "0.0966666666666602" 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG } 05:51:47 INFO [http_cmd]: http status code OK 05:51:47 DEBUG --- status code --- 05:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:47 DEBUG --- http response --- 05:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:47 DEBUG ----- output ----- 05:51:47 DEBUG { 05:51:47 DEBUG "status": "success", 05:51:47 DEBUG "data": { 05:51:47 DEBUG "resultType": "vector", 05:51:47 DEBUG "result": [ 05:51:47 DEBUG { 05:51:47 DEBUG "metric": { 05:51:47 DEBUG "pod": "ds-idrepo-0" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG [...] 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG "pod": "ds-idrepo-2" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 1713588707, 05:51:47 DEBUG "0" 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG } 05:51:47 INFO [http_cmd]: http status code OK 05:51:47 DEBUG --- status code --- 05:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:47 DEBUG --- http response --- 05:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:47 DEBUG ----- output ----- 05:51:47 DEBUG { 05:51:47 DEBUG "status": "success", 05:51:47 DEBUG "data": { 05:51:47 DEBUG "resultType": "vector", 05:51:47 DEBUG "result": [ 05:51:47 DEBUG { 05:51:47 DEBUG "metric": { 05:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG [...] 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG 05:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:47 DEBUG }, 05:51:47 DEBUG "value": [ 05:51:47 DEBUG 1713588707, 05:51:47 DEBUG "0" 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG ] 05:51:47 DEBUG } 05:51:47 DEBUG } 05:51:48 INFO 05:51:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:51:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:48 INFO 05:51:48 INFO [loop_until]: kubectl --namespace=xlou top pods 05:51:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588708" 05:51:48 INFO [loop_until]: OK (rc = 0) 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [loop_until]: OK (rc = 0) 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- stdout --- 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- status code --- 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- stdout --- 05:51:48 DEBUG --- status code --- 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17306m 8653Mi am-767f6b67b5-qcqz2 17283m 8714Mi am-767f6b67b5-vbltz 18998m 8650Mi ds-cts-0 18915m 10626Mi ds-cts-1 17120m 10717Mi ds-cts-2 19102m 10610Mi ds-idrepo-0 7643m 3714Mi ds-idrepo-1 7586m 3701Mi ds-idrepo-2 7888m 3588Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7803m 7377Mi 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- status code --- 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 INFO [http_cmd]: http status code OK 05:51:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6212Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7895m 26% 7392Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17306m 57% 11356Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3372Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18405m 61% 13769Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17001m 56% 11599Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18801m 62% 2992Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19381m 64% 10861Mi 9% gke-xlou-cdm-ds-0f601751-j488 8231m 27% 6904Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17200m 57% 15716Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7732m 25% 7599Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18445m 61% 13201Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8183m 51% 11189Mi 19% 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG --- status code --- 05:51:48 DEBUG --- stderr --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- stderr --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG --- http response --- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:48 DEBUG 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG { 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:48 DEBUG 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG { 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG ----- output ----- 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG "data": { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "data": { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "status": "success", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "data": { 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "data": { 05:51:48 DEBUG "data": { 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "data": { 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG { 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "resultType": "vector", 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "metric": { 05:51:48 DEBUG { 05:51:48 DEBUG "result": [ 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG "result": [ 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "pod": "ds-cts-0" 05:51:48 DEBUG "metric": { 05:51:48 DEBUG { 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:48 DEBUG "pod": "ds-idrepo-0" 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "pod": "ds-cts-0" 05:51:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "instance": "10.142.0.107:9100" 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "instance": "10.142.0.107:9100" 05:51:48 DEBUG "metric": { 05:51:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:51:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:48 DEBUG }, 05:51:48 DEBUG }, 05:51:48 DEBUG "metric": { 05:51:48 DEBUG { 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG "instance": "10.142.0.107:9100" 05:51:48 DEBUG "pod": "ds-cts-0" 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "ds-idrepo-0" 05:51:48 DEBUG }, 05:51:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "pod": "ds-idrepo-0" 05:51:48 DEBUG "metric": { 05:51:48 DEBUG }, 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG }, 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "am-767f6b67b5-9zw22" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "pod": "ds-cts-2" 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG [...] 05:51:48 DEBUG 05:51:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:48 DEBUG 05:51:48 DEBUG "pod": "ds-idrepo-0" 05:51:48 DEBUG "pod": "ds-cts-2" 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "ds-idrepo-0" 05:51:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:48 DEBUG }, 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "pod": "ds-cts-2" 05:51:48 DEBUG }, 05:51:48 DEBUG "instance": "10.142.0.43:9100" 05:51:48 DEBUG 05:51:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:51:48 DEBUG }, 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:48 DEBUG }, 05:51:48 DEBUG 05:51:48 DEBUG 05:51:48 DEBUG "pod": "ds-cts-2" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "instance": "10.142.0.43:9100" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "instance": "10.142.0.43:9100" 05:51:48 DEBUG 05:51:48 DEBUG }, 05:51:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "pod": "ds-idrepo-2" 05:51:48 DEBUG 05:51:48 DEBUG }, 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG }, 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "ds-idrepo-2" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG }, 05:51:48 DEBUG "pod": "am-767f6b67b5-vbltz" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "2662.633333333333" 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "0.0005666666666666667" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "0" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "2663.4666666666667" 05:51:48 DEBUG "5546.002301807595" 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "30" 05:51:48 DEBUG "value": [ 05:51:48 DEBUG }, 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG ] 05:51:48 DEBUG "0" 05:51:48 DEBUG "479797.5" 05:51:48 DEBUG ] 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG ] 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "80500822.4" 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG "8.029817415788495" 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG "0.092" 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG "value": [ 05:51:48 DEBUG "0" 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG "0.25979999999999465" 05:51:48 DEBUG } 05:51:48 DEBUG "3099314299904" 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG ] 05:51:48 DEBUG "7835222016" 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG 1713588708, 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG "0" 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG "0" 05:51:48 DEBUG "2663.0333333333333" 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG ] 05:51:48 DEBUG ] 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:51:48 DEBUG } 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588767" 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588767" 05:52:47 ERROR [http_cmd]: ERROR 05:52:47 ERROR --- status code --- 05:52:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:52:47 ERROR --- http response --- 05:52:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:52:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588767" 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:47 DEBUG ----- output ----- 05:52:47 DEBUG { 05:52:47 DEBUG "status": "success", 05:52:47 DEBUG "data": { 05:52:47 DEBUG "resultType": "vector", 05:52:47 DEBUG "result": [ 05:52:47 DEBUG { 05:52:47 DEBUG "metric": { 05:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:47 DEBUG }, 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG [...] 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:47 DEBUG }, 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 1713588767, 05:52:47 DEBUG "2661.2" 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588767" 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:52:47 DEBUG ----- output ----- 05:52:47 DEBUG { 05:52:47 DEBUG "status": "success", 05:52:47 DEBUG "data": { 05:52:47 DEBUG "resultType": "vector", 05:52:47 DEBUG "result": [ 05:52:47 DEBUG { 05:52:47 DEBUG "metric": { 05:52:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:47 DEBUG }, 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG [...] 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:47 DEBUG }, 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 1713588767, 05:52:47 DEBUG "17455.499098508444" 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588767" 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588767" 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588767" 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588767" 05:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588767" 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 DEBUG ----- output ----- 05:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:47 DEBUG ----- output ----- 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG { 05:52:47 DEBUG ----- output ----- 05:52:47 DEBUG { 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG "status": "success", 05:52:47 INFO [http_cmd]: http status code OK 05:52:47 DEBUG { 05:52:47 DEBUG "status": "success", 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG "data": { 05:52:47 DEBUG --- status code --- 05:52:47 DEBUG "status": "success", 05:52:47 DEBUG "data": { 05:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:47 DEBUG "resultType": "vector", 05:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:47 DEBUG "data": { 05:52:47 DEBUG "resultType": "vector", 05:52:47 DEBUG ----- output ----- 05:52:47 DEBUG "result": [ 05:52:47 DEBUG --- http response --- 05:52:47 DEBUG "resultType": "vector", 05:52:47 DEBUG "result": [ 05:52:47 DEBUG { 05:52:47 DEBUG { 05:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:47 DEBUG "result": [ 05:52:47 DEBUG { 05:52:47 DEBUG "status": "success", 05:52:47 DEBUG "metric": { 05:52:47 DEBUG ----- output ----- 05:52:47 DEBUG { 05:52:47 DEBUG "metric": { 05:52:47 DEBUG "data": { 05:52:47 DEBUG "pod": "ds-idrepo-0" 05:52:47 DEBUG { 05:52:47 DEBUG "metric": { 05:52:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:47 DEBUG "resultType": "vector", 05:52:47 DEBUG "result": [ 05:52:47 DEBUG { 05:52:47 DEBUG "metric": { 05:52:47 DEBUG }, 05:52:47 DEBUG "status": "success", 05:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:47 DEBUG "instance": "10.142.0.107:9100" 05:52:47 DEBUG }, 05:52:47 DEBUG "value": [ 05:52:47 DEBUG "data": { 05:52:47 DEBUG }, 05:52:47 DEBUG }, 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 05:52:47 DEBUG "resultType": "vector", 05:52:47 DEBUG "value": [ 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG "result": [ 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG { 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG [...] 05:52:47 DEBUG "metric": { 05:52:47 DEBUG 05:52:47 DEBUG [...] 05:52:47 DEBUG [...] 05:52:47 DEBUG 05:52:47 DEBUG "instance": "10.142.0.107:9100" 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG [...] 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG }, 05:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG 05:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:47 DEBUG }, 05:52:47 DEBUG "pod": "ds-idrepo-2" 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 05:52:47 DEBUG "instance": "10.142.0.43:9100" 05:52:47 DEBUG "value": [ 05:52:47 DEBUG }, 05:52:47 DEBUG }, 05:52:47 DEBUG 05:52:47 DEBUG }, 05:52:47 DEBUG 1713588767, 05:52:47 DEBUG "value": [ 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 05:52:47 DEBUG "value": [ 05:52:47 DEBUG "17.285296836584916" 05:52:47 DEBUG 1713588767, 05:52:47 DEBUG 1713588767, 05:52:47 DEBUG [...] 05:52:47 DEBUG 1713588767, 05:52:47 DEBUG ] 05:52:47 DEBUG "0" 05:52:47 DEBUG "0" 05:52:47 DEBUG 05:52:47 DEBUG "80618819.83333333" 05:52:47 DEBUG } 05:52:47 DEBUG ] 05:52:47 DEBUG ] 05:52:47 DEBUG 05:52:47 DEBUG ] 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 DEBUG 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 DEBUG ] 05:52:47 DEBUG ] 05:52:47 DEBUG "instance": "10.142.0.43:9100" 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 DEBUG }, 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:47 DEBUG "value": [ 05:52:47 DEBUG 1713588767, 05:52:47 DEBUG "0.10533333333333035" 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG ] 05:52:47 DEBUG } 05:52:47 DEBUG } 05:52:48 INFO 05:52:48 INFO [loop_until]: kubectl --namespace=xlou top pods 05:52:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:48 INFO 05:52:48 INFO [loop_until]: kubectl --namespace=xlou top node 05:52:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588769" 05:52:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588769" 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [loop_until]: OK (rc = 0) 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [loop_until]: OK (rc = 0) 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- stdout --- 05:52:49 DEBUG --- status code --- 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- stdout --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17041m 8653Mi am-767f6b67b5-qcqz2 17162m 8712Mi am-767f6b67b5-vbltz 19085m 8649Mi ds-cts-0 18950m 10619Mi ds-cts-1 17872m 10718Mi ds-cts-2 19001m 10601Mi ds-idrepo-0 7839m 3714Mi ds-idrepo-1 7601m 3701Mi ds-idrepo-2 8067m 3588Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1176Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 16m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8272m 7377Mi 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6209Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7976m 26% 7386Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17190m 57% 11367Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3367Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19113m 64% 13762Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17443m 58% 11589Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19126m 64% 3006Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19544m 65% 10863Mi 9% gke-xlou-cdm-ds-0f601751-j488 8271m 27% 6899Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17628m 59% 15718Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8030m 26% 7581Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18979m 63% 13199Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8167m 51% 11188Mi 19% 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- stderr --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 INFO [http_cmd]: http status code OK 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:49 DEBUG --- stderr --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG --- status code --- 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:52:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:52:49 DEBUG { 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG --- http response --- 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:52:49 DEBUG "data": { 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG { 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "data": { 05:52:49 DEBUG "data": { 05:52:49 DEBUG "data": { 05:52:49 DEBUG { 05:52:49 DEBUG ----- output ----- 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG { 05:52:49 DEBUG "data": { 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "data": { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "data": { 05:52:49 DEBUG "status": "success", 05:52:49 DEBUG { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "result": [ 05:52:49 DEBUG { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "result": [ 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "data": { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "metric": { 05:52:49 DEBUG { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "resultType": "vector", 05:52:49 DEBUG "instance": "10.142.0.107:9100" 05:52:49 DEBUG { 05:52:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:49 DEBUG "metric": { 05:52:49 DEBUG { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:52:49 DEBUG { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:49 DEBUG "pod": "ds-cts-0" 05:52:49 DEBUG "pod": "ds-cts-0" 05:52:49 DEBUG "pod": "ds-idrepo-0" 05:52:49 DEBUG { 05:52:49 DEBUG "metric": { 05:52:49 DEBUG }, 05:52:49 DEBUG "metric": { 05:52:49 DEBUG }, 05:52:49 DEBUG "pod": "ds-idrepo-0" 05:52:49 DEBUG "metric": { 05:52:49 DEBUG { 05:52:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:49 DEBUG "instance": "10.142.0.107:9100" 05:52:49 DEBUG }, 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "pod": "ds-cts-0" 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG "pod": "ds-idrepo-0" 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:49 DEBUG "value": [ 05:52:49 DEBUG }, 05:52:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:52:49 DEBUG "metric": { 05:52:49 DEBUG "result": [ 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "instance": "10.142.0.107:9100" 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG }, 05:52:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:52:49 DEBUG { 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG }, 05:52:49 DEBUG "metric": { 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG "pod": "ds-cts-2" 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "pod": "ds-cts-2" 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:49 DEBUG "pod": "ds-cts-2" 05:52:49 DEBUG "pod": "ds-idrepo-2" 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG [...] 05:52:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:49 DEBUG "instance": "10.142.0.43:9100" 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "pod": "ds-cts-2" 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "instance": "10.142.0.43:9100" 05:52:49 DEBUG }, 05:52:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:52:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:49 DEBUG "pod": "ds-idrepo-2" 05:52:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:52:49 DEBUG 05:52:49 DEBUG 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG }, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG "0" 05:52:49 DEBUG "value": [ 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG }, 05:52:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:52:49 DEBUG 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "instance": "10.142.0.43:9100" 05:52:49 DEBUG "7832584192" 05:52:49 DEBUG "value": [ 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG "0" 05:52:49 DEBUG ] 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "value": [ 05:52:49 DEBUG }, 05:52:49 DEBUG 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG }, 05:52:49 DEBUG ] 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG "0.092" 05:52:49 DEBUG "0.0006666666666666668" 05:52:49 DEBUG "0" 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG "3103644957696" 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG "2668.366666666667" 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG "value": [ 05:52:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:52:49 DEBUG "2669.4" 05:52:49 DEBUG "479738.5" 05:52:49 DEBUG "80614326.13333333" 05:52:49 DEBUG "value": [ 05:52:49 DEBUG } 05:52:49 DEBUG "30" 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG "0" 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG ] 05:52:49 DEBUG "0" 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG }, 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG "0" 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG "2668.233333333333" 05:52:49 DEBUG "value": [ 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG "0.28600000000009135" 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG 1713588769, 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG "8.057601811166654" 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG ] 05:52:49 DEBUG } 05:52:49 DEBUG } 05:52:49 DEBUG } 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588827" 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:47 DEBUG ----- output ----- 05:53:47 DEBUG { 05:53:47 DEBUG "status": "success", 05:53:47 DEBUG "data": { 05:53:47 DEBUG "resultType": "vector", 05:53:47 DEBUG "result": [ 05:53:47 DEBUG { 05:53:47 DEBUG "metric": { 05:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG [...] 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 1713588827, 05:53:47 DEBUG "2665.866666666667" 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588827" 05:53:47 ERROR [http_cmd]: ERROR 05:53:47 ERROR --- status code --- 05:53:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:53:47 ERROR --- http response --- 05:53:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:53:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588827" 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588827" 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:47 DEBUG ----- output ----- 05:53:47 DEBUG { 05:53:47 DEBUG "status": "success", 05:53:47 DEBUG "data": { 05:53:47 DEBUG "resultType": "vector", 05:53:47 DEBUG "result": [ 05:53:47 DEBUG { 05:53:47 DEBUG "metric": { 05:53:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG [...] 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 1713588827, 05:53:47 DEBUG "490946392.3753573" 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588827" 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588827" 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588827" 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588827" 05:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588827" 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:47 DEBUG ----- output ----- 05:53:47 DEBUG { 05:53:47 DEBUG "status": "success", 05:53:47 DEBUG "data": { 05:53:47 DEBUG "resultType": "vector", 05:53:47 DEBUG "result": [ 05:53:47 DEBUG { 05:53:47 DEBUG "metric": { 05:53:47 DEBUG "pod": "ds-idrepo-0" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG [...] 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG "pod": "ds-idrepo-2" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 1713588827, 05:53:47 DEBUG "0" 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:47 DEBUG ----- output ----- 05:53:47 DEBUG ----- output ----- 05:53:47 DEBUG { 05:53:47 DEBUG { 05:53:47 DEBUG "status": "success", 05:53:47 DEBUG "status": "success", 05:53:47 DEBUG "data": { 05:53:47 DEBUG "data": { 05:53:47 DEBUG "resultType": "vector", 05:53:47 DEBUG "resultType": "vector", 05:53:47 DEBUG "result": [ 05:53:47 DEBUG "result": [ 05:53:47 DEBUG { 05:53:47 DEBUG { 05:53:47 DEBUG "metric": { 05:53:47 DEBUG "metric": { 05:53:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:47 DEBUG "instance": "10.142.0.107:9100" 05:53:47 DEBUG }, 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG [...] 05:53:47 DEBUG [...] 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:53:47 DEBUG "instance": "10.142.0.43:9100" 05:53:47 DEBUG }, 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 1713588827, 05:53:47 DEBUG 1713588827, 05:53:47 DEBUG "15.99481996305145" 05:53:47 DEBUG "80641794.43333334" 05:53:47 DEBUG ] 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 DEBUG ] 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:47 DEBUG ----- output ----- 05:53:47 DEBUG { 05:53:47 DEBUG "status": "success", 05:53:47 DEBUG "data": { 05:53:47 DEBUG "resultType": "vector", 05:53:47 DEBUG "result": [ 05:53:47 DEBUG { 05:53:47 DEBUG "metric": { 05:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG [...] 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 1713588827, 05:53:47 DEBUG "0" 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:47 INFO [http_cmd]: http status code OK 05:53:47 DEBUG --- status code --- 05:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:47 DEBUG --- http response --- 05:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:47 DEBUG ----- output ----- 05:53:47 DEBUG { 05:53:47 DEBUG "status": "success", 05:53:47 DEBUG "data": { 05:53:47 DEBUG "resultType": "vector", 05:53:47 DEBUG "result": [ 05:53:47 DEBUG { 05:53:47 DEBUG "metric": { 05:53:47 DEBUG "instance": "10.142.0.107:9100" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG [...] 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG 05:53:47 DEBUG "instance": "10.142.0.43:9100" 05:53:47 DEBUG }, 05:53:47 DEBUG "value": [ 05:53:47 DEBUG 1713588827, 05:53:47 DEBUG "0.06599999999999491" 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG ] 05:53:47 DEBUG } 05:53:47 DEBUG } 05:53:49 INFO 05:53:49 INFO [loop_until]: kubectl --namespace=xlou top pods 05:53:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:49 INFO 05:53:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:53:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588829" 05:53:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588829" 05:53:49 INFO [loop_until]: OK (rc = 0) 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [loop_until]: OK (rc = 0) 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG --- stdout --- 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG --- status code --- 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 INFO [http_cmd]: http status code OK 05:53:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 16988m 8652Mi am-767f6b67b5-qcqz2 17033m 8713Mi am-767f6b67b5-vbltz 18738m 8650Mi ds-cts-0 18798m 10624Mi ds-cts-1 16698m 10742Mi ds-cts-2 19467m 10602Mi ds-idrepo-0 7728m 3714Mi ds-idrepo-1 7500m 3704Mi ds-idrepo-2 7892m 3588Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8048m 7377Mi 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- stdout --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- status code --- 05:53:49 DEBUG --- stderr --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 6204Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8000m 26% 7381Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17274m 57% 11370Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3373Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18515m 62% 13750Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5201Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17402m 58% 11595Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19250m 64% 3009Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19457m 65% 10868Mi 9% gke-xlou-cdm-ds-0f601751-j488 8272m 27% 6913Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16918m 56% 15715Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8049m 26% 7598Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19592m 65% 13205Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8179m 51% 11188Mi 19% 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:53:49 DEBUG 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG { 05:53:49 DEBUG --- stderr --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG { 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG --- http response --- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG { 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG { 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG { 05:53:49 DEBUG "data": { 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG "data": { 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG ----- output ----- 05:53:49 DEBUG { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "status": "success", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "data": { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "instance": "10.142.0.107:9100" 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "resultType": "vector", 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG }, 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:49 DEBUG { 05:53:49 DEBUG { 05:53:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:49 DEBUG "pod": "ds-cts-0" 05:53:49 DEBUG "pod": "ds-idrepo-0" 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "result": [ 05:53:49 DEBUG "value": [ 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "result": [ 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG { 05:53:49 DEBUG }, 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG }, 05:53:49 DEBUG }, 05:53:49 DEBUG }, 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "pod": "ds-cts-0" 05:53:49 DEBUG { 05:53:49 DEBUG 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "instance": "10.142.0.107:9100" 05:53:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:53:49 DEBUG "pod": "ds-cts-0" 05:53:49 DEBUG { 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:53:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "pod": "ds-idrepo-0" 05:53:49 DEBUG }, 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG }, 05:53:49 DEBUG }, 05:53:49 DEBUG }, 05:53:49 DEBUG "instance": "10.142.0.107:9100" 05:53:49 DEBUG }, 05:53:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG }, 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG "metric": { 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "value": [ 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG }, 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "pod": "ds-idrepo-0" 05:53:49 DEBUG 05:53:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG }, 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG [...] 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "instance": "10.142.0.43:9100" 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:53:49 DEBUG "pod": "ds-cts-2" 05:53:49 DEBUG }, 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "pod": "ds-cts-2" 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG }, 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG [...] 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 05:53:49 DEBUG "instance": "10.142.0.43:9100" 05:53:49 DEBUG 05:53:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:49 DEBUG 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG 05:53:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG "pod": "ds-cts-2" 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "80636730.53333333" 05:53:49 DEBUG 05:53:49 DEBUG "pod": "ds-cts-2" 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG "instance": "10.142.0.43:9100" 05:53:49 DEBUG }, 05:53:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:49 DEBUG "2667.9666666666667" 05:53:49 DEBUG }, 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:53:49 DEBUG "0.0021765533333333335" 05:53:49 DEBUG "0" 05:53:49 DEBUG }, 05:53:49 DEBUG "pod": "ds-idrepo-2" 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG ] 05:53:49 DEBUG 05:53:49 DEBUG }, 05:53:49 DEBUG "0" 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG }, 05:53:49 DEBUG ] 05:53:49 DEBUG "value": [ 05:53:49 DEBUG "0" 05:53:49 DEBUG }, 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG "value": [ 05:53:49 DEBUG }, 05:53:49 DEBUG "2670.0333333333333" 05:53:49 DEBUG } 05:53:49 DEBUG 05:53:49 DEBUG "value": [ 05:53:49 DEBUG ] 05:53:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG } 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG ] 05:53:49 DEBUG "value": [ 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG "pod": "ds-idrepo-2" 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG } 05:53:49 DEBUG }, 05:53:49 DEBUG "3107356836864" 05:53:49 DEBUG ] 05:53:49 DEBUG "0.092" 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG ] 05:53:49 DEBUG "7832268800" 05:53:49 DEBUG } 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG "0" 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG }, 05:53:49 DEBUG "value": [ 05:53:49 DEBUG ] 05:53:49 DEBUG "value": [ 05:53:49 DEBUG } 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG ] 05:53:49 DEBUG "2668.0666666666666" 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG "479847.5" 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG "0" 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG "29.966666666666665" 05:53:49 DEBUG } 05:53:49 DEBUG 1713588829, 05:53:49 DEBUG ] 05:53:49 DEBUG "0.1765333333332535" 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG "8.038291727312883" 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG "0" 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG ] 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:53:49 DEBUG } 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588887" 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:47 DEBUG ----- output ----- 05:54:47 DEBUG { 05:54:47 DEBUG "status": "success", 05:54:47 DEBUG "data": { 05:54:47 DEBUG "resultType": "vector", 05:54:47 DEBUG "result": [ 05:54:47 DEBUG { 05:54:47 DEBUG "metric": { 05:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG [...] 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 1713588887, 05:54:47 DEBUG "2670.2" 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG } 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588887" 05:54:47 ERROR [http_cmd]: ERROR 05:54:47 ERROR --- status code --- 05:54:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:54:47 ERROR --- http response --- 05:54:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:54:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588887" 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588887" 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:54:47 DEBUG ----- output ----- 05:54:47 DEBUG { 05:54:47 DEBUG "status": "success", 05:54:47 DEBUG "data": { 05:54:47 DEBUG "resultType": "vector", 05:54:47 DEBUG "result": [ 05:54:47 DEBUG { 05:54:47 DEBUG "metric": { 05:54:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG [...] 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 1713588887, 05:54:47 DEBUG "438765236.51163876" 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG } 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588887" 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588887" 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588887" 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588887" 05:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588887" 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:47 DEBUG ----- output ----- 05:54:47 DEBUG { 05:54:47 DEBUG "status": "success", 05:54:47 DEBUG "data": { 05:54:47 DEBUG "resultType": "vector", 05:54:47 DEBUG "result": [ 05:54:47 DEBUG { 05:54:47 DEBUG "metric": { 05:54:47 DEBUG "pod": "ds-idrepo-0" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG [...] 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG "pod": "ds-idrepo-2" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 1713588887, 05:54:47 DEBUG "0" 05:54:47 DEBUG ] 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG } 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG } 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:47 DEBUG ----- output ----- 05:54:47 DEBUG { 05:54:47 DEBUG "status": "success", 05:54:47 DEBUG "data": { 05:54:47 DEBUG "resultType": "vector", 05:54:47 DEBUG "result": [ 05:54:47 DEBUG { 05:54:47 DEBUG "metric": { 05:54:47 DEBUG "instance": "10.142.0.107:9100" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG [...] 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG "instance": "10.142.0.43:9100" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 1713588887, 05:54:47 DEBUG "80671495.33333333" 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG } 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:54:47 DEBUG ----- output ----- 05:54:47 DEBUG { 05:54:47 DEBUG "status": "success", 05:54:47 DEBUG "data": { 05:54:47 DEBUG "resultType": "vector", 05:54:47 DEBUG "result": [ 05:54:47 DEBUG { 05:54:47 DEBUG "metric": { 05:54:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:47 DEBUG }, 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG [...] 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:54:47 DEBUG }, 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG "value": [ 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG 1713588887, 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG "16.00393233188801" 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG ] 05:54:47 INFO [http_cmd]: http status code OK 05:54:47 DEBUG } 05:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:47 DEBUG --- status code --- 05:54:47 DEBUG ] 05:54:47 DEBUG ----- output ----- 05:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:47 DEBUG } 05:54:47 DEBUG { 05:54:47 DEBUG --- http response --- 05:54:47 DEBUG } 05:54:47 DEBUG "status": "success", 05:54:47 DEBUG "data": { 05:54:47 DEBUG "resultType": "vector", 05:54:47 DEBUG "result": [ 05:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:47 DEBUG { 05:54:47 DEBUG ----- output ----- 05:54:47 DEBUG "metric": { 05:54:47 DEBUG { 05:54:47 DEBUG "instance": "10.142.0.107:9100" 05:54:47 DEBUG "status": "success", 05:54:47 DEBUG }, 05:54:47 DEBUG "data": { 05:54:47 DEBUG "resultType": "vector", 05:54:47 DEBUG "value": [ 05:54:47 DEBUG "result": [ 05:54:47 DEBUG 05:54:47 DEBUG { 05:54:47 DEBUG 05:54:47 DEBUG "metric": { 05:54:47 DEBUG 05:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:47 DEBUG [...] 05:54:47 DEBUG }, 05:54:47 DEBUG 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG 05:54:47 DEBUG "instance": "10.142.0.43:9100" 05:54:47 DEBUG 05:54:47 DEBUG }, 05:54:47 DEBUG [...] 05:54:47 DEBUG "value": [ 05:54:47 DEBUG 05:54:47 DEBUG 1713588887, 05:54:47 DEBUG 05:54:47 DEBUG "0.05333333333333219" 05:54:47 DEBUG 05:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:47 DEBUG ] 05:54:47 DEBUG }, 05:54:47 DEBUG } 05:54:47 DEBUG "value": [ 05:54:47 DEBUG ] 05:54:47 DEBUG 1713588887, 05:54:47 DEBUG } 05:54:47 DEBUG "0" 05:54:47 DEBUG } 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG ] 05:54:47 DEBUG } 05:54:47 DEBUG } 05:54:49 INFO 05:54:49 INFO [loop_until]: kubectl --namespace=xlou top pods 05:54:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:49 INFO 05:54:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:54:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588889" 05:54:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588889" 05:54:49 INFO [loop_until]: OK (rc = 0) 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [loop_until]: OK (rc = 0) 05:54:49 DEBUG --- stdout --- 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- status code --- 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- status code --- 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- status code --- 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- status code --- 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- stdout --- 05:54:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17270m 8651Mi am-767f6b67b5-qcqz2 17353m 8713Mi am-767f6b67b5-vbltz 18944m 8650Mi ds-cts-0 18951m 10648Mi ds-cts-1 16786m 10727Mi ds-cts-2 18445m 10600Mi ds-idrepo-0 7883m 3714Mi ds-idrepo-1 7489m 3704Mi ds-idrepo-2 7871m 3593Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8221m 7377Mi 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6194Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7955m 26% 7377Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17372m 58% 11361Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18462m 61% 13765Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17463m 58% 11591Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18636m 62% 3007Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19531m 65% 10863Mi 9% gke-xlou-cdm-ds-0f601751-j488 8271m 27% 6912Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17263m 57% 15707Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8076m 27% 7606Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19096m 63% 13203Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8192m 51% 11184Mi 19% 05:54:49 DEBUG --- stderr --- 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG --- http response --- 05:54:49 INFO [http_cmd]: http status code OK 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG --- stderr --- 05:54:49 DEBUG 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:54:49 DEBUG --- status code --- 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG { 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "data": { 05:54:49 DEBUG --- http response --- 05:54:49 DEBUG { 05:54:49 DEBUG "data": { 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "data": { 05:54:49 DEBUG "data": { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "result": [ 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "data": { 05:54:49 DEBUG { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "result": [ 05:54:49 DEBUG { 05:54:49 DEBUG "data": { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "data": { 05:54:49 DEBUG ----- output ----- 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "data": { 05:54:49 DEBUG "data": { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "data": { 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG { 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "data": { 05:54:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "data": { 05:54:49 DEBUG { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "result": [ 05:54:49 DEBUG { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "status": "success", 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "result": [ 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:49 DEBUG { 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "result": [ 05:54:49 DEBUG 05:54:49 DEBUG { 05:54:49 DEBUG "data": { 05:54:49 DEBUG { 05:54:49 DEBUG "pod": "ds-cts-0" 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "result": [ 05:54:49 DEBUG { 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:49 DEBUG { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "instance": "10.142.0.107:9100" 05:54:49 DEBUG }, 05:54:49 DEBUG "pod": "ds-cts-0" 05:54:49 DEBUG }, 05:54:49 DEBUG { 05:54:49 DEBUG { 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG "resultType": "vector", 05:54:49 DEBUG "metric": { 05:54:49 DEBUG }, 05:54:49 DEBUG "instance": "10.142.0.107:9100" 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG { 05:54:49 DEBUG }, 05:54:49 DEBUG }, 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "result": [ 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "pod": "ds-idrepo-0" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "result": [ 05:54:49 DEBUG "pod": "ds-idrepo-0" 05:54:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:49 DEBUG 05:54:49 DEBUG { 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "instance": "10.142.0.107:9100" 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG { 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG "metric": { 05:54:49 DEBUG 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "pod": "ds-idrepo-0" 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "metric": { 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG "metric": { 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG [...] 05:54:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:54:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "pod": "ds-cts-0" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG [...] 05:54:49 DEBUG [...] 05:54:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG "7830503424" 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG ] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG "instance": "10.142.0.43:9100" 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "pod": "ds-cts-2" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "pod": "ds-cts-2" 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG "instance": "10.142.0.43:9100" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:49 DEBUG }, 05:54:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG } 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG }, 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG "pod": "ds-cts-2" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:49 DEBUG "value": [ 05:54:49 DEBUG }, 05:54:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG ] 05:54:49 DEBUG "0.13740000000011318" 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG }, 05:54:49 DEBUG [...] 05:54:49 DEBUG "pod": "ds-idrepo-2" 05:54:49 DEBUG }, 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG }, 05:54:49 DEBUG [...] 05:54:49 DEBUG 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:54:49 DEBUG "0" 05:54:49 DEBUG "0" 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG "instance": "10.142.0.43:9100" 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG "0.0006999999999999999" 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "0.092" 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 05:54:49 DEBUG "pod": "ds-idrepo-2" 05:54:49 DEBUG ] 05:54:49 DEBUG }, 05:54:49 DEBUG ] 05:54:49 DEBUG ] 05:54:49 DEBUG 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG "3111279311872" 05:54:49 DEBUG }, 05:54:49 DEBUG ] 05:54:49 DEBUG 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG ] 05:54:49 DEBUG "2668.733333333333" 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG 05:54:49 DEBUG }, 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "value": [ 05:54:49 DEBUG ] 05:54:49 DEBUG "value": [ 05:54:49 DEBUG } 05:54:49 DEBUG 05:54:49 DEBUG "479942" 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG "2671.366666666667" 05:54:49 DEBUG 05:54:49 DEBUG "value": [ 05:54:49 DEBUG } 05:54:49 DEBUG }, 05:54:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:54:49 DEBUG } 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG } 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG ] 05:54:49 DEBUG 05:54:49 DEBUG ] 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:54:49 DEBUG }, 05:54:49 DEBUG "value": [ 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG ] 05:54:49 DEBUG "0" 05:54:49 DEBUG ] 05:54:49 DEBUG "80669697" 05:54:49 DEBUG "0" 05:54:49 DEBUG } 05:54:49 DEBUG "pod": "ds-cts-2" 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG ] 05:54:49 DEBUG }, 05:54:49 DEBUG "0" 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG }, 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG "8.041254174970778" 05:54:49 DEBUG } 05:54:49 DEBUG "value": [ 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG "value": [ 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG "0" 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG 1713588889, 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG "2670.6" 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG "29.966666666666665" 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG ] 05:54:49 DEBUG } 05:54:49 DEBUG } 05:54:49 DEBUG } 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588947" 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:47 DEBUG ----- output ----- 05:55:47 DEBUG { 05:55:47 DEBUG "status": "success", 05:55:47 DEBUG "data": { 05:55:47 DEBUG "resultType": "vector", 05:55:47 DEBUG "result": [ 05:55:47 DEBUG { 05:55:47 DEBUG "metric": { 05:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:47 DEBUG }, 05:55:47 DEBUG[0m "value": [ 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG [...] 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:47 DEBUG }, 05:55:47 DEBUG "value": [ 05:55:47 DEBUG 1713588947, 05:55:47 DEBUG "2668.8333333333335" 05:55:47 DEBUG ] 05:55:47 DEBUG } 05:55:47 DEBUG ] 05:55:47 DEBUG } 05:55:47 DEBUG } 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713588947" 05:55:47 ERROR [http_cmd]: ERROR 05:55:47 ERROR --- status code --- 05:55:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:55:47 ERROR --- http response --- 05:55:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:55:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588947" 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588947" 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:55:47 DEBUG ----- output ----- 05:55:47 DEBUG { 05:55:47 DEBUG "status": "success", 05:55:47 DEBUG "data": { 05:55:47 DEBUG "resultType": "vector", 05:55:47 DEBUG "result": [ 05:55:47 DEBUG { 05:55:47 DEBUG "metric": { 05:55:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:47 DEBUG }, 05:55:47 DEBUG "value": [ 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG [...] 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG "pod": "ds-cts-0" 05:55:47 DEBUG }, 05:55:47 DEBUG "value": [ 05:55:47 DEBUG 1713588947, 05:55:47 DEBUG "320968164.5178968" 05:55:47 DEBUG ] 05:55:47 DEBUG } 05:55:47 DEBUG ] 05:55:47 DEBUG } 05:55:47 DEBUG } 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588947" 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588947" 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588947" 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588947" 05:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588947" 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:47 DEBUG ----- output ----- 05:55:47 DEBUG ----- output ----- 05:55:47 DEBUG { 05:55:47 DEBUG { 05:55:47 DEBUG "status": "success", 05:55:47 DEBUG "status": "success", 05:55:47 DEBUG "data": { 05:55:47 DEBUG "data": { 05:55:47 DEBUG "resultType": "vector", 05:55:47 DEBUG "resultType": "vector", 05:55:47 DEBUG "result": [ 05:55:47 DEBUG "result": [ 05:55:47 DEBUG { 05:55:47 DEBUG { 05:55:47 DEBUG "metric": { 05:55:47 DEBUG "metric": { 05:55:47 DEBUG "pod": "ds-idrepo-0" 05:55:47 DEBUG "instance": "10.142.0.107:9100" 05:55:47 DEBUG }, 05:55:47 DEBUG }, 05:55:47 DEBUG "value": [ 05:55:47 DEBUG "value": [ 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 05:55:47 DEBUG [...] 05:55:47 DEBUG [...] 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG ----- output ----- 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG { 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG "status": "success", 05:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 INFO [http_cmd]: http status code OK 05:55:47 DEBUG "data": { 05:55:47 DEBUG ----- output ----- 05:55:47 DEBUG "instance": "10.142.0.43:9100" 05:55:47 DEBUG "pod": "ds-idrepo-2" 05:55:47 DEBUG --- status code --- 05:55:47 DEBUG "resultType": "vector", 05:55:47 DEBUG { 05:55:47 DEBUG }, 05:55:47 DEBUG "value": [ 05:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:47 DEBUG "result": [ 05:55:47 DEBUG { 05:55:47 DEBUG }, 05:55:47 DEBUG 1713588947, 05:55:47 DEBUG --- http response --- 05:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:47 DEBUG "status": "success", 05:55:47 DEBUG "value": [ 05:55:47 DEBUG "81115509.36666666" 05:55:47 DEBUG ----- output ----- 05:55:47 DEBUG "metric": { 05:55:47 DEBUG "data": { 05:55:47 DEBUG 1713588947, 05:55:47 DEBUG ] 05:55:47 DEBUG { 05:55:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:47 DEBUG "resultType": "vector", 05:55:47 DEBUG "0" 05:55:47 DEBUG } 05:55:47 DEBUG "status": "success", 05:55:47 DEBUG }, 05:55:47 DEBUG "result": [ 05:55:47 DEBUG ] 05:55:47 DEBUG ] 05:55:47 DEBUG "data": { 05:55:47 DEBUG "value": [ 05:55:47 DEBUG { 05:55:47 DEBUG } 05:55:47 DEBUG } 05:55:47 DEBUG "resultType": "vector", 05:55:47 DEBUG 05:55:47 DEBUG "metric": { 05:55:47 DEBUG ] 05:55:47 DEBUG } 05:55:47 DEBUG "result": [ 05:55:47 DEBUG 05:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:47 DEBUG } 05:55:47 DEBUG { 05:55:47 DEBUG 05:55:47 DEBUG }, 05:55:47 DEBUG } 05:55:47 DEBUG "metric": { 05:55:47 DEBUG [...] 05:55:47 DEBUG "value": [ 05:55:47 DEBUG "instance": "10.142.0.107:9100" 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG }, 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG [...] 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG "value": [ 05:55:47 DEBUG "pod": "ds-cts-0" 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG }, 05:55:47 DEBUG 05:55:47 DEBUG 05:55:47 DEBUG "value": [ 05:55:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:47 DEBUG 05:55:47 DEBUG 1713588947, 05:55:47 DEBUG }, 05:55:47 DEBUG [...] 05:55:47 DEBUG "19.24009287300555" 05:55:47 DEBUG "value": [ 05:55:47 DEBUG 05:55:47 DEBUG ] 05:55:47 DEBUG 1713588947, 05:55:47 DEBUG 05:55:47 DEBUG } 05:55:47 DEBUG "0" 05:55:47 DEBUG 05:55:47 DEBUG ] 05:55:47 DEBUG ] 05:55:47 DEBUG "instance": "10.142.0.43:9100" 05:55:47 DEBUG } 05:55:47 DEBUG } 05:55:47 DEBUG }, 05:55:47 DEBUG } 05:55:47 DEBUG ] 05:55:47 DEBUG "value": [ 05:55:47 DEBUG 1713588947, 05:55:47 DEBUG } 05:55:47 DEBUG "0.10766666666667486" 05:55:47 DEBUG } 05:55:47 DEBUG ] 05:55:47 DEBUG } 05:55:47 DEBUG ] 05:55:47 DEBUG } 05:55:47 DEBUG } 05:55:49 INFO 05:55:49 INFO [loop_until]: kubectl --namespace=xlou top pods 05:55:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:49 INFO 05:55:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:55:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "result": [ 05:55:49 DEBUG { 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG "2668.866666666667" 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713588949" 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [loop_until]: OK (rc = 0) 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713588949" 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG --- stdout --- 05:55:49 DEBUG --- status code --- 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG --- status code --- 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17444m 8648Mi am-767f6b67b5-qcqz2 17253m 8712Mi am-767f6b67b5-vbltz 18324m 8650Mi ds-cts-0 19659m 10618Mi ds-cts-1 17446m 10719Mi ds-cts-2 19376m 10594Mi ds-idrepo-0 7772m 3714Mi ds-idrepo-1 7450m 3704Mi ds-idrepo-2 8021m 3594Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8066m 7378Mi 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- status code --- 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG --- stderr --- 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:49 DEBUG 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:49 DEBUG --- http response --- 05:55:49 INFO [http_cmd]: http status code OK 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:49 DEBUG { 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 INFO [loop_until]: OK (rc = 0) 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:49 DEBUG --- status code --- 05:55:49 DEBUG { 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:55:49 DEBUG --- stdout --- 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG { 05:55:49 DEBUG { 05:55:49 DEBUG { 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG { 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG { 05:55:49 DEBUG "data": { 05:55:49 DEBUG { 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6204Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7971m 26% 7382Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17376m 58% 11354Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3375Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19209m 64% 13763Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5188Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17482m 58% 11597Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19183m 64% 3008Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19397m 64% 10865Mi 9% gke-xlou-cdm-ds-0f601751-j488 8000m 26% 6909Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17598m 58% 15714Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8019m 26% 7600Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18865m 63% 13208Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8194m 51% 11186Mi 19% 05:55:49 DEBUG { 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG { 05:55:49 DEBUG --- http response --- 05:55:49 DEBUG "data": { 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG { 05:55:49 DEBUG --- stderr --- 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "data": { 05:55:49 DEBUG "data": { 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "data": { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG 05:55:49 DEBUG "data": { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG ----- output ----- 05:55:49 DEBUG "data": { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "data": { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "data": { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG { 05:55:49 DEBUG { 05:55:49 DEBUG "status": "success", 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "metric": { 05:55:49 DEBUG { 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "result": [ 05:55:49 DEBUG { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "pod": "ds-cts-0" 05:55:49 DEBUG "data": { 05:55:49 DEBUG { 05:55:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:55:49 DEBUG { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG { 05:55:49 DEBUG { 05:55:49 DEBUG "metric": { 05:55:49 DEBUG { 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "metric": { 05:55:49 DEBUG { 05:55:49 DEBUG { 05:55:49 DEBUG }, 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "resultType": "vector", 05:55:49 DEBUG "metric": { 05:55:49 DEBUG }, 05:55:49 DEBUG "metric": { 05:55:49 DEBUG { 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:55:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:49 DEBUG "instance": "10.142.0.107:9100" 05:55:49 DEBUG "instance": "10.142.0.107:9100" 05:55:49 DEBUG { 05:55:49 DEBUG "pod": "ds-idrepo-0" 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "result": [ 05:55:49 DEBUG "pod": "ds-cts-0" 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:49 DEBUG }, 05:55:49 DEBUG }, 05:55:49 DEBUG }, 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "metric": { 05:55:49 DEBUG }, 05:55:49 DEBUG }, 05:55:49 DEBUG "metric": { 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG }, 05:55:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:49 DEBUG 05:55:49 DEBUG "pod": "ds-cts-0" 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG { 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG }, 05:55:49 DEBUG 05:55:49 DEBUG }, 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG { 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "metric": { 05:55:49 DEBUG "pod": "ds-idrepo-0" 05:55:49 DEBUG { 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG [...] 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG "metric": { 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG }, 05:55:49 DEBUG [...] 05:55:49 DEBUG "pod": "ds-idrepo-0" 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "metric": { 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "instance": "10.142.0.107:9100" 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG }, 05:55:49 DEBUG "pod": "ds-cts-2" 05:55:49 DEBUG }, 05:55:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG }, 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG }, 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:49 DEBUG 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:55:49 DEBUG }, 05:55:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:49 DEBUG }, 05:55:49 DEBUG }, 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "instance": "10.142.0.43:9100" 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG }, 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:49 DEBUG }, 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "pod": "ds-cts-0" 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG [...] 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "pod": "ds-idrepo-2" 05:55:49 DEBUG 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG 05:55:49 DEBUG "pod": "ds-idrepo-2" 05:55:49 DEBUG "0.0010011466666666666" 05:55:49 DEBUG 05:55:49 DEBUG [...] 05:55:49 DEBUG }, 05:55:49 DEBUG "7833006080" 05:55:49 DEBUG 05:55:49 DEBUG "instance": "10.142.0.43:9100" 05:55:49 DEBUG "2668.9333333333334" 05:55:49 DEBUG "8.027154260275681" 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG }, 05:55:49 DEBUG 05:55:49 DEBUG "2669.4666666666667" 05:55:49 DEBUG "pod": "ds-cts-2" 05:55:49 DEBUG }, 05:55:49 DEBUG ] 05:55:49 DEBUG 05:55:49 DEBUG 05:55:49 DEBUG "value": [ 05:55:49 DEBUG ] 05:55:49 DEBUG "pod": "ds-cts-2" 05:55:49 DEBUG }, 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG "3115617039360" 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG "value": [ 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG } 05:55:49 DEBUG }, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG } 05:55:49 DEBUG 05:55:49 DEBUG "instance": "10.142.0.43:9100" 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG }, 05:55:49 DEBUG ] 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG } 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG "pod": "ds-cts-2" 05:55:49 DEBUG 05:55:49 DEBUG "78848856.50832108" 05:55:49 DEBUG ] 05:55:49 DEBUG "value": [ 05:55:49 DEBUG "0.2790333333332446" 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG }, 05:55:49 DEBUG "0.092" 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG "value": [ 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG "0" 05:55:49 DEBUG }, 05:55:49 DEBUG "pod": "ds-cts-0" 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG "value": [ 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG "480252.5" 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG "value": [ 05:55:49 DEBUG }, 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG "30" 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG } 05:55:49 DEBUG "0" 05:55:49 DEBUG "0" 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG "value": [ 05:55:49 DEBUG 1713588949, 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG "81110623.30000001" 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG "0" 05:55:49 DEBUG ] 05:55:49 DEBUG "61145715.554786414" 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG ] 05:55:49 DEBUG ] 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:55:49 DEBUG } 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589007" 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:47 DEBUG ----- output ----- 05:56:47 DEBUG { 05:56:47 DEBUG "status": "success", 05:56:47 DEBUG "data": { 05:56:47 DEBUG "resultType": "vector", 05:56:47 DEBUG "result": [ 05:56:47 DEBUG { 05:56:47 DEBUG "metric": { 05:56:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:56:47 DEBUG }, 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG [...] 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:56:47 DEBUG }, 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 1713589007, 05:56:47 DEBUG "2670.233333333333" 05:56:47 DEBUG ] 05:56:47 DEBUG } 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589007" 05:56:47 DEBUG ] 05:56:47 DEBUG } 05:56:47 DEBUG } 05:56:47 ERROR [http_cmd]: ERROR 05:56:47 ERROR --- status code --- 05:56:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:56:47 ERROR --- http response --- 05:56:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:56:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589007" 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589007" 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:47 DEBUG ----- output ----- 05:56:47 DEBUG { 05:56:47 DEBUG "status": "success", 05:56:47 DEBUG "data": { 05:56:47 DEBUG "resultType": "vector", 05:56:47 DEBUG "result": [ 05:56:47 DEBUG { 05:56:47 DEBUG "metric": { 05:56:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:56:47 DEBUG }, 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG [...] 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:56:47 DEBUG }, 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 1713589007, 05:56:47 DEBUG "29026.330227565464" 05:56:47 DEBUG ] 05:56:47 DEBUG } 05:56:47 DEBUG ] 05:56:47 DEBUG } 05:56:47 DEBUG } 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589007" 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589007" 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589007" 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589007" 05:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589007" 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:47 DEBUG ----- output ----- 05:56:47 DEBUG { 05:56:47 DEBUG "status": "success", 05:56:47 DEBUG "data": { 05:56:47 DEBUG "resultType": "vector", 05:56:47 DEBUG "result": [ 05:56:47 DEBUG { 05:56:47 DEBUG "metric": { 05:56:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:56:47 DEBUG }, 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG [...] 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG }, 05:56:47 DEBUG "value": [ 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG 1713589007, 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG "16.005439103133924" 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG ] 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:47 DEBUG } 05:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:47 INFO [http_cmd]: http status code OK 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG ----- output ----- 05:56:47 DEBUG ] 05:56:47 DEBUG ----- output ----- 05:56:47 DEBUG --- status code --- 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG { 05:56:47 DEBUG } 05:56:47 DEBUG { 05:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:47 DEBUG "status": "success", 05:56:47 DEBUG } 05:56:47 DEBUG "status": "success", 05:56:47 DEBUG --- http response --- 05:56:47 DEBUG ----- output ----- 05:56:47 DEBUG "data": { 05:56:47 DEBUG "data": { 05:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:47 DEBUG { 05:56:47 DEBUG "resultType": "vector", 05:56:47 DEBUG "resultType": "vector", 05:56:47 DEBUG ----- output ----- 05:56:47 DEBUG "status": "success", 05:56:47 DEBUG "result": [ 05:56:47 DEBUG "result": [ 05:56:47 DEBUG { 05:56:47 DEBUG "data": { 05:56:47 DEBUG { 05:56:47 DEBUG { 05:56:47 DEBUG "status": "success", 05:56:47 DEBUG "resultType": "vector", 05:56:47 DEBUG "metric": { 05:56:47 DEBUG "metric": { 05:56:47 DEBUG "data": { 05:56:47 DEBUG "resultType": "vector", 05:56:47 DEBUG "pod": "ds-idrepo-0" 05:56:47 DEBUG "instance": "10.142.0.107:9100" 05:56:47 DEBUG }, 05:56:47 DEBUG }, 05:56:47 DEBUG "result": [ 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 05:56:47 DEBUG "result": [ 05:56:47 DEBUG { 05:56:47 DEBUG 05:56:47 DEBUG "value": [ 05:56:47 DEBUG { 05:56:47 DEBUG "metric": { 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG "metric": { 05:56:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:56:47 DEBUG [...] 05:56:47 DEBUG 05:56:47 DEBUG "instance": "10.142.0.107:9100" 05:56:47 DEBUG }, 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG }, 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 05:56:47 DEBUG [...] 05:56:47 DEBUG "value": [ 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG "instance": "10.142.0.43:9100" 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG }, 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG [...] 05:56:47 DEBUG "value": [ 05:56:47 DEBUG "pod": "ds-idrepo-2" 05:56:47 DEBUG [...] 05:56:47 DEBUG 05:56:47 DEBUG 1713589007, 05:56:47 DEBUG "80918399.76666667" 05:56:47 DEBUG ] 05:56:47 DEBUG 05:56:47 DEBUG }, 05:56:47 DEBUG } 05:56:47 DEBUG 05:56:47 DEBUG 05:56:47 DEBUG "value": [ 05:56:47 DEBUG ] 05:56:47 DEBUG 05:56:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:56:47 DEBUG 1713589007, 05:56:47 DEBUG } 05:56:47 DEBUG 05:56:47 DEBUG }, 05:56:47 DEBUG "0" 05:56:47 DEBUG } 05:56:47 DEBUG "instance": "10.142.0.43:9100" 05:56:47 DEBUG "value": [ 05:56:47 DEBUG ] 05:56:47 DEBUG }, 05:56:47 DEBUG 1713589007, 05:56:47 DEBUG } 05:56:47 DEBUG "value": [ 05:56:47 DEBUG "0" 05:56:47 DEBUG ] 05:56:47 DEBUG 1713589007, 05:56:47 DEBUG ] 05:56:47 DEBUG } 05:56:47 DEBUG "0.06866666666667431" 05:56:47 DEBUG } 05:56:47 DEBUG } 05:56:47 DEBUG ] 05:56:47 DEBUG ] 05:56:47 DEBUG } 05:56:47 DEBUG } 05:56:47 DEBUG ] 05:56:47 DEBUG } 05:56:47 DEBUG } 05:56:47 DEBUG } 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG { 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "data": { 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "result": [ 05:56:49 DEBUG { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG "2680.366666666667" 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 INFO 05:56:49 INFO [loop_until]: kubectl --namespace=xlou top pods 05:56:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:49 INFO 05:56:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:56:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589009" 05:56:49 INFO [loop_until]: OK (rc = 0) 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- status code --- 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- status code --- 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [loop_until]: OK (rc = 0) 05:56:49 DEBUG --- stdout --- 05:56:49 DEBUG --- stdout --- 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- status code --- 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- http response --- 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6202Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7995m 26% 7390Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17345m 58% 11362Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3380Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19114m 64% 13756Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5217Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17516m 58% 11602Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19224m 64% 3014Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19529m 65% 10863Mi 9% gke-xlou-cdm-ds-0f601751-j488 8249m 27% 6932Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17240m 57% 15712Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8079m 27% 7589Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19519m 65% 13199Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 7924m 49% 11183Mi 19% 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17410m 8651Mi am-767f6b67b5-qcqz2 17340m 8712Mi am-767f6b67b5-vbltz 18805m 8649Mi ds-cts-0 19019m 10601Mi ds-cts-1 17607m 10720Mi ds-cts-2 19573m 10580Mi ds-idrepo-0 7784m 3714Mi ds-idrepo-1 7694m 3704Mi ds-idrepo-2 7992m 3618Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 17m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8141m 7377Mi 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- stderr --- 05:56:49 DEBUG 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- stderr --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG { 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG ----- output ----- 05:56:49 INFO [http_cmd]: http status code OK 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "data": { 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG { 05:56:49 DEBUG --- status code --- 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:56:49 DEBUG { 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:49 DEBUG { 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG { 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:49 DEBUG "data": { 05:56:49 DEBUG --- http response --- 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:56:49 DEBUG { 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "result": [ 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "data": { 05:56:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:56:49 DEBUG { 05:56:49 DEBUG "data": { 05:56:49 DEBUG "data": { 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG { 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "data": { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG ----- output ----- 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG { 05:56:49 DEBUG "data": { 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "data": { 05:56:49 DEBUG "data": { 05:56:49 DEBUG "data": { 05:56:49 DEBUG "data": { 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG { 05:56:49 DEBUG "result": [ 05:56:49 DEBUG { 05:56:49 DEBUG "data": { 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "data": { 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "data": { 05:56:49 DEBUG "pod": "ds-cts-0" 05:56:49 DEBUG "data": { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "result": [ 05:56:49 DEBUG { 05:56:49 DEBUG "data": { 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG { 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG }, 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "pod": "ds-cts-0" 05:56:49 DEBUG "instance": "10.142.0.107:9100" 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG "status": "success", 05:56:49 DEBUG "data": { 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG { 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "result": [ 05:56:49 DEBUG }, 05:56:49 DEBUG }, 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "resultType": "vector", 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "instance": "10.142.0.107:9100" 05:56:49 DEBUG { 05:56:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:56:49 DEBUG { 05:56:49 DEBUG 05:56:49 DEBUG { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "pod": "ds-idrepo-0" 05:56:49 DEBUG "instance": "10.142.0.107:9100" 05:56:49 DEBUG "result": [ 05:56:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:56:49 DEBUG { 05:56:49 DEBUG "pod": "ds-cts-0" 05:56:49 DEBUG "pod": "ds-idrepo-0" 05:56:49 DEBUG "pod": "ds-idrepo-1" 05:56:49 DEBUG }, 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "metric": { 05:56:49 DEBUG }, 05:56:49 DEBUG "metric": { 05:56:49 DEBUG 05:56:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG { 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG }, 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG "metric": { 05:56:49 DEBUG 05:56:49 DEBUG "metric": { 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG "pod": "ds-idrepo-1" 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "pod": "ds-idrepo-0" 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG [...] 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "pod": "ds-cts-2" 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "instance": "10.142.0.43:9100" 05:56:49 DEBUG 05:56:49 DEBUG "pod": "ds-cts-2" 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "0.0006" 05:56:49 DEBUG [...] 05:56:49 DEBUG "3119445173248" 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG "pod": "ds-cts-2" 05:56:49 DEBUG 05:56:49 DEBUG "instance": "10.142.0.43:9100" 05:56:49 DEBUG }, 05:56:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG ] 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:56:49 DEBUG "value": [ 05:56:49 DEBUG }, 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "pod": "ds-idrepo-2" 05:56:49 DEBUG ] 05:56:49 DEBUG 05:56:49 DEBUG } 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG [...] 05:56:49 DEBUG 05:56:49 DEBUG "instance": "10.142.0.43:9100" 05:56:49 DEBUG "2681.6" 05:56:49 DEBUG "pod": "ds-cts-2" 05:56:49 DEBUG }, 05:56:49 DEBUG }, 05:56:49 DEBUG }, 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 05:56:49 DEBUG ] 05:56:49 DEBUG "0" 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG ] 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "value": [ 05:56:49 DEBUG "8.04869640440969" 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG 05:56:49 DEBUG }, 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG } 05:56:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG 05:56:49 DEBUG 05:56:49 DEBUG "value": [ 05:56:49 DEBUG } 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG ] 05:56:49 DEBUG "2680" 05:56:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG ] 05:56:49 DEBUG }, 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG 05:56:49 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG ] 05:56:49 DEBUG "0" 05:56:49 DEBUG "32.1" 05:56:49 DEBUG ] 05:56:49 DEBUG "0" 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG } 05:56:49 DEBUG "0" 05:56:49 DEBUG "value": [ 05:56:49 DEBUG ] 05:56:49 DEBUG "pod": "ds-idrepo-2" 05:56:49 DEBUG }, 05:56:49 DEBUG "0.17256666666663514" 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG "80914061.9" 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG } 05:56:49 DEBUG }, 05:56:49 DEBUG "value": [ 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG ] 05:56:49 DEBUG "0.092" 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG "7832535040" 05:56:49 DEBUG ] 05:56:49 DEBUG "480229" 05:56:49 DEBUG } 05:56:49 DEBUG "value": [ 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG 1713589009, 05:56:49 DEBUG "0" 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG "0" 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG ] 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:56:49 DEBUG } 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589067" 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589067" 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:47 DEBUG ----- output ----- 05:57:47 DEBUG { 05:57:47 DEBUG "status": "success", 05:57:47 DEBUG "data": { 05:57:47 DEBUG "resultType": "vector", 05:57:47 DEBUG "result": [ 05:57:47 DEBUG { 05:57:47 DEBUG "metric": { 05:57:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG [...] 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 1713589067, 05:57:47 DEBUG "2666.2" 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG } 05:57:47 ERROR [http_cmd]: ERROR 05:57:47 ERROR --- status code --- 05:57:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:57:47 ERROR --- http response --- 05:57:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:57:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589067" 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589067" 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:47 DEBUG ----- output ----- 05:57:47 DEBUG { 05:57:47 DEBUG "status": "success", 05:57:47 DEBUG "data": { 05:57:47 DEBUG "resultType": "vector", 05:57:47 DEBUG "result": [ 05:57:47 DEBUG { 05:57:47 DEBUG "metric": { 05:57:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG [...] 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 1713589067, 05:57:47 DEBUG "220915672.53215638" 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG } 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589067" 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589067" 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:47 DEBUG ----- output ----- 05:57:47 DEBUG { 05:57:47 DEBUG "status": "success", 05:57:47 DEBUG "data": { 05:57:47 DEBUG "resultType": "vector", 05:57:47 DEBUG "result": [ 05:57:47 DEBUG { 05:57:47 DEBUG "metric": { 05:57:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG [...] 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 1713589067, 05:57:47 DEBUG "0" 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG } 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589067" 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:47 DEBUG ----- output ----- 05:57:47 DEBUG { 05:57:47 DEBUG "status": "success", 05:57:47 DEBUG "data": { 05:57:47 DEBUG "resultType": "vector", 05:57:47 DEBUG "result": [ 05:57:47 DEBUG { 05:57:47 DEBUG "metric": { 05:57:47 DEBUG "instance": "10.142.0.107:9100" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG [...] 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG "instance": "10.142.0.43:9100" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 1713589067, 05:57:47 DEBUG "0.10200000000000199" 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG } 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589067" 05:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589067" 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:47 DEBUG ----- output ----- 05:57:47 DEBUG { 05:57:47 DEBUG "status": "success", 05:57:47 DEBUG "data": { 05:57:47 DEBUG "resultType": "vector", 05:57:47 DEBUG "result": [ 05:57:47 DEBUG { 05:57:47 DEBUG "metric": { 05:57:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG [...] 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 1713589067, 05:57:47 DEBUG "15.999822866778993" 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG } 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:47 DEBUG ----- output ----- 05:57:47 DEBUG { 05:57:47 DEBUG "status": "success", 05:57:47 DEBUG "data": { 05:57:47 DEBUG "resultType": "vector", 05:57:47 DEBUG "result": [ 05:57:47 DEBUG { 05:57:47 DEBUG "metric": { 05:57:47 DEBUG "instance": "10.142.0.107:9100" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 INFO [http_cmd]: http status code OK 05:57:47 DEBUG --- status code --- 05:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:47 DEBUG --- http response --- 05:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:47 DEBUG ----- output ----- 05:57:47 DEBUG { 05:57:47 DEBUG "status": "success", 05:57:47 DEBUG "data": { 05:57:47 DEBUG "resultType": "vector", 05:57:47 DEBUG "result": [ 05:57:47 DEBUG { 05:57:47 DEBUG "metric": { 05:57:47 DEBUG "pod": "ds-idrepo-0" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG [...] 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG "pod": "ds-idrepo-2" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 1713589067, 05:57:47 DEBUG "0" 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG } 05:57:47 DEBUG 05:57:47 DEBUG [...] 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG 05:57:47 DEBUG "instance": "10.142.0.43:9100" 05:57:47 DEBUG }, 05:57:47 DEBUG "value": [ 05:57:47 DEBUG 1713589067, 05:57:47 DEBUG "80662425.86666667" 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG ] 05:57:47 DEBUG } 05:57:47 DEBUG } 05:57:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589069" 05:57:49 INFO [http_cmd]: http status code OK 05:57:49 DEBUG --- status code --- 05:57:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:49 DEBUG --- http response --- 05:57:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:49 DEBUG ----- output ----- 05:57:49 DEBUG { 05:57:49 DEBUG "status": "success", 05:57:49 DEBUG "data": { 05:57:49 DEBUG "resultType": "vector", 05:57:49 DEBUG "result": [ 05:57:49 DEBUG { 05:57:49 DEBUG "metric": { 05:57:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:57:49 DEBUG }, 05:57:49 DEBUG "value": [ 05:57:49 DEBUG 05:57:49 DEBUG 05:57:49 DEBUG 05:57:49 DEBUG [...] 05:57:49 DEBUG 05:57:49 DEBUG 05:57:49 DEBUG 05:57:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:57:49 DEBUG }, 05:57:49 DEBUG "value": [ 05:57:49 DEBUG 1713589069, 05:57:49 DEBUG "2664.233333333333" 05:57:49 DEBUG ] 05:57:49 DEBUG } 05:57:49 DEBUG ] 05:57:49 DEBUG } 05:57:49 DEBUG } 05:57:49 INFO 05:57:49 INFO [loop_until]: kubectl --namespace=xlou top node 05:57:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:57:49 INFO 05:57:49 INFO [loop_until]: kubectl --namespace=xlou top pods 05:57:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589070" 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [loop_until]: OK (rc = 0) 05:57:50 DEBUG --- stdout --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- status code --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- status code --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- status code --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- status code --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17906m 8651Mi am-767f6b67b5-qcqz2 17295m 8716Mi am-767f6b67b5-vbltz 18695m 8651Mi ds-cts-0 18452m 10631Mi ds-cts-1 16698m 10744Mi ds-cts-2 18743m 10591Mi ds-idrepo-0 7528m 3714Mi ds-idrepo-1 7710m 3704Mi ds-idrepo-2 7865m 3627Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 18m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7950m 7377Mi 05:57:50 DEBUG --- status code --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG --- status code --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 INFO [loop_until]: OK (rc = 0) 05:57:50 DEBUG --- stderr --- 05:57:50 DEBUG 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG --- http response --- 05:57:50 INFO [http_cmd]: http status code OK 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG --- stdout --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:50 DEBUG --- status code --- 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6208Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7951m 26% 7373Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17193m 57% 11361Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3377Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18521m 62% 13760Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 5193Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17420m 58% 11609Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19042m 63% 3002Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 18704m 62% 10855Mi 9% gke-xlou-cdm-ds-0f601751-j488 8286m 27% 6950Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17266m 57% 15714Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8077m 27% 7583Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18872m 63% 13203Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8188m 51% 11185Mi 19% 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG { 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:50 DEBUG { 05:57:50 DEBUG --- stderr --- 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:50 DEBUG { 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG { 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG { 05:57:50 DEBUG --- http response --- 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG { 05:57:50 DEBUG { 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG "data": { 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG { 05:57:50 DEBUG { 05:57:50 DEBUG { 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:57:50 DEBUG { 05:57:50 DEBUG "data": { 05:57:50 DEBUG { 05:57:50 DEBUG { 05:57:50 DEBUG "data": { 05:57:50 DEBUG { 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "data": { 05:57:50 DEBUG { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG { 05:57:50 DEBUG "data": { 05:57:50 DEBUG ----- output ----- 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "data": { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "data": { 05:57:50 DEBUG "data": { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "data": { 05:57:50 DEBUG "data": { 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG { 05:57:50 DEBUG "data": { 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "data": { 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG { 05:57:50 DEBUG { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "data": { 05:57:50 DEBUG "data": { 05:57:50 DEBUG "result": [ 05:57:50 DEBUG { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "data": { 05:57:50 DEBUG "result": [ 05:57:50 DEBUG { 05:57:50 DEBUG "result": [ 05:57:50 DEBUG { 05:57:50 DEBUG "data": { 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "instance": "10.142.0.107:9100" 05:57:50 DEBUG }, 05:57:50 DEBUG { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "status": "success", 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "result": [ 05:57:50 DEBUG { 05:57:50 DEBUG { 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "result": [ 05:57:50 DEBUG { 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "pod": "ds-idrepo-0" 05:57:50 DEBUG }, 05:57:50 DEBUG "result": [ 05:57:50 DEBUG { 05:57:50 DEBUG { 05:57:50 DEBUG "pod": "ds-idrepo-0" 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:57:50 DEBUG "pod": "ds-idrepo-0" 05:57:50 DEBUG { 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG "result": [ 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "instance": "10.142.0.107:9100" 05:57:50 DEBUG "metric": { 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "metric": { 05:57:50 DEBUG }, 05:57:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:57:50 DEBUG }, 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "pod": "ds-cts-0" 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG { 05:57:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG }, 05:57:50 DEBUG "pod": "ds-cts-0" 05:57:50 DEBUG 05:57:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:57:50 DEBUG 05:57:50 DEBUG "pod": "ds-idrepo-2" 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "result": [ 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG { 05:57:50 DEBUG "data": { 05:57:50 DEBUG "resultType": "vector", 05:57:50 DEBUG "result": [ 05:57:50 DEBUG { 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "pod": "ds-cts-2" 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "pod": "ds-cts-2" 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG "480679" 05:57:50 DEBUG ] 05:57:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG "metric": { 05:57:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG } 05:57:50 DEBUG [...] 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG "instance": "10.142.0.107:9100" 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG }, 05:57:50 DEBUG }, 05:57:50 DEBUG }, 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG "0" 05:57:50 DEBUG 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG ] 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "value": [ 05:57:50 DEBUG ] 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG 05:57:50 DEBUG "pod": "ds-idrepo-2" 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG } 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 05:57:50 DEBUG "result": [ 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG } 05:57:50 DEBUG "0" 05:57:50 DEBUG "instance": "10.142.0.43:9100" 05:57:50 DEBUG 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG } 05:57:50 DEBUG 05:57:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:57:50 DEBUG { 05:57:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "0" 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG [...] 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG }, 05:57:50 DEBUG "metric": { 05:57:50 DEBUG }, 05:57:50 DEBUG "0" 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:57:50 DEBUG "0" 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "pod": "ds-cts-0" 05:57:50 DEBUG "value": [ 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG [...] 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG }, 05:57:50 DEBUG ] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "pod": "ds-cts-2" 05:57:50 DEBUG [...] 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG ] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG "0.2830666666666512" 05:57:50 DEBUG "value": [ 05:57:50 DEBUG } 05:57:50 DEBUG }, 05:57:50 DEBUG "instance": "10.142.0.43:9100" 05:57:50 DEBUG 05:57:50 DEBUG "8.039462722851624" 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG 05:57:50 DEBUG } 05:57:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:57:50 DEBUG 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG ] 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG ] 05:57:50 DEBUG "value": [ 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG ] 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG 05:57:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:57:50 DEBUG }, 05:57:50 DEBUG 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG "2664.4666666666667" 05:57:50 DEBUG } 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG 05:57:50 DEBUG } 05:57:50 DEBUG [...] 05:57:50 DEBUG 05:57:50 DEBUG }, 05:57:50 DEBUG "value": [ 05:57:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG "3123072095232" 05:57:50 DEBUG 05:57:50 DEBUG ] 05:57:50 DEBUG 05:57:50 DEBUG "7832309760" 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG }, 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG "30" 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG 05:57:50 DEBUG ] 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG "0.092" 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG "pod": "ds-cts-2" 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG ] 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG "instance": "10.142.0.43:9100" 05:57:50 DEBUG }, 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG "2663.366666666667" 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG }, 05:57:50 DEBUG } 05:57:50 DEBUG "0" 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG "value": [ 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG "0.000033333333333333335" 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG ] 05:57:50 DEBUG "value": [ 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG 1713589070, 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG } 05:57:50 DEBUG "80656168.16666666" 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG ] 05:57:50 DEBUG } 05:57:50 DEBUG } 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589127" 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589127" 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:47 DEBUG ----- output ----- 05:58:47 DEBUG { 05:58:47 DEBUG "status": "success", 05:58:47 DEBUG "data": { 05:58:47 DEBUG "resultType": "vector", 05:58:47 DEBUG "result": [ 05:58:47 DEBUG { 05:58:47 DEBUG "metric": { 05:58:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG [...] 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 1713589127, 05:58:47 DEBUG "2660.0333333333333" 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG } 05:58:47 ERROR [http_cmd]: ERROR 05:58:47 ERROR --- status code --- 05:58:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:58:47 ERROR --- http response --- 05:58:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:58:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589127" 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589127" 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:47 DEBUG ----- output ----- 05:58:47 DEBUG { 05:58:47 DEBUG "status": "success", 05:58:47 DEBUG "data": { 05:58:47 DEBUG "resultType": "vector", 05:58:47 DEBUG "result": [ 05:58:47 DEBUG { 05:58:47 DEBUG "metric": { 05:58:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG [...] 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 1713589127, 05:58:47 DEBUG "100891.23833089642" 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG } 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589127" 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589127" 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:47 DEBUG ----- output ----- 05:58:47 DEBUG { 05:58:47 DEBUG "status": "success", 05:58:47 DEBUG "data": { 05:58:47 DEBUG "resultType": "vector", 05:58:47 DEBUG "result": [ 05:58:47 DEBUG { 05:58:47 DEBUG "metric": { 05:58:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG [...] 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 1713589127, 05:58:47 DEBUG "0" 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG } 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589127" 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:47 DEBUG ----- output ----- 05:58:47 DEBUG { 05:58:47 DEBUG "status": "success", 05:58:47 DEBUG "data": { 05:58:47 DEBUG "resultType": "vector", 05:58:47 DEBUG "result": [ 05:58:47 DEBUG { 05:58:47 DEBUG "metric": { 05:58:47 DEBUG "instance": "10.142.0.107:9100" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG [...] 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG "instance": "10.142.0.43:9100" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 1713589127, 05:58:47 DEBUG "0.09833333333333485" 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG } 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589127" 05:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589127" 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:58:47 DEBUG ----- output ----- 05:58:47 DEBUG { 05:58:47 DEBUG "status": "success", 05:58:47 DEBUG "data": { 05:58:47 DEBUG "resultType": "vector", 05:58:47 DEBUG "result": [ 05:58:47 DEBUG { 05:58:47 DEBUG "metric": { 05:58:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG [...] 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 1713589127, 05:58:47 DEBUG "17.30706365987929" 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG } 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:47 DEBUG ----- output ----- 05:58:47 DEBUG { 05:58:47 DEBUG "status": "success", 05:58:47 DEBUG "data": { 05:58:47 DEBUG "resultType": "vector", 05:58:47 DEBUG "result": [ 05:58:47 DEBUG { 05:58:47 DEBUG "metric": { 05:58:47 DEBUG "pod": "ds-idrepo-0" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG [...] 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG "pod": "ds-idrepo-2" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 1713589127, 05:58:47 DEBUG "0" 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG } 05:58:47 INFO [http_cmd]: http status code OK 05:58:47 DEBUG --- status code --- 05:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:47 DEBUG --- http response --- 05:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:47 DEBUG ----- output ----- 05:58:47 DEBUG { 05:58:47 DEBUG "status": "success", 05:58:47 DEBUG "data": { 05:58:47 DEBUG "resultType": "vector", 05:58:47 DEBUG "result": [ 05:58:47 DEBUG { 05:58:47 DEBUG "metric": { 05:58:47 DEBUG "instance": "10.142.0.107:9100" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG [...] 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG 05:58:47 DEBUG "instance": "10.142.0.43:9100" 05:58:47 DEBUG }, 05:58:47 DEBUG "value": [ 05:58:47 DEBUG 1713589127, 05:58:47 DEBUG "81094879.53333333" 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG ] 05:58:47 DEBUG } 05:58:47 DEBUG } 05:58:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589129" 05:58:49 INFO [http_cmd]: http status code OK 05:58:49 DEBUG --- status code --- 05:58:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:49 DEBUG --- http response --- 05:58:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:49 DEBUG ----- output ----- 05:58:49 DEBUG { 05:58:49 DEBUG "status": "success", 05:58:49 DEBUG "data": { 05:58:49 DEBUG "resultType": "vector", 05:58:49 DEBUG "result": [ 05:58:49 DEBUG { 05:58:49 DEBUG "metric": { 05:58:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:49 DEBUG }, 05:58:49 DEBUG "value": [ 05:58:49 DEBUG 05:58:49 DEBUG 05:58:49 DEBUG 05:58:49 DEBUG [...] 05:58:49 DEBUG 05:58:49 DEBUG 05:58:49 DEBUG 05:58:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:49 DEBUG }, 05:58:49 DEBUG "value": [ 05:58:49 DEBUG 1713589129, 05:58:49 DEBUG "2663.5666666666666" 05:58:49 DEBUG ] 05:58:49 DEBUG } 05:58:49 DEBUG ] 05:58:49 DEBUG } 05:58:49 DEBUG } 05:58:50 INFO 05:58:50 INFO [loop_until]: kubectl --namespace=xlou top pods 05:58:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:58:50 INFO 05:58:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:58:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589130" 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589130" 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG --- status code --- 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589130" 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- http response --- 05:58:50 INFO [loop_until]: OK (rc = 0) 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- http response --- 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG --- status code --- 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589130" 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG --- stdout --- 05:58:50 DEBUG --- status code --- 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG --- status code --- 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17153m 8650Mi am-767f6b67b5-qcqz2 17223m 8710Mi am-767f6b67b5-vbltz 18580m 8649Mi ds-cts-0 18617m 10635Mi ds-cts-1 17408m 10721Mi ds-cts-2 18755m 10597Mi ds-idrepo-0 7620m 3714Mi ds-idrepo-1 7703m 3705Mi ds-idrepo-2 7838m 3626Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 14m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7818m 7377Mi 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG --- status code --- 05:58:50 INFO [loop_until]: OK (rc = 0) 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG { 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- stderr --- 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- stdout --- 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG --- http response --- 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6212Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7971m 26% 7387Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17255m 57% 11356Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3382Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18757m 62% 13779Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5198Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 16844m 56% 11599Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19138m 64% 3024Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19340m 64% 10856Mi 9% gke-xlou-cdm-ds-0f601751-j488 8254m 27% 6952Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 16806m 56% 15716Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8080m 27% 7601Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18750m 62% 13208Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8191m 51% 11183Mi 19% 05:58:50 DEBUG --- stderr --- 05:58:50 INFO [http_cmd]: http status code OK 05:58:50 DEBUG { 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:50 DEBUG "data": { 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG { 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:50 DEBUG --- status code --- 05:58:50 DEBUG 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG { 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:58:50 DEBUG "data": { 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG "result": [ 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG "data": { 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG --- http response --- 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "data": { 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:58:50 DEBUG "data": { 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "data": { 05:58:50 DEBUG "data": { 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "data": { 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "data": { 05:58:50 DEBUG "data": { 05:58:50 DEBUG ----- output ----- 05:58:50 DEBUG { 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "data": { 05:58:50 DEBUG "data": { 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "data": { 05:58:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG { 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "data": { 05:58:50 DEBUG "data": { 05:58:50 DEBUG { 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG { 05:58:50 DEBUG "metric": { 05:58:50 DEBUG { 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG }, 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG { 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "status": "success", 05:58:50 DEBUG "pod": "ds-cts-0" 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "result": [ 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "value": [ 05:58:50 DEBUG { 05:58:50 DEBUG "data": { 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "result": [ 05:58:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:58:50 DEBUG }, 05:58:50 DEBUG { 05:58:50 DEBUG "data": { 05:58:50 DEBUG }, 05:58:50 DEBUG "instance": "10.142.0.107:9100" 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "metric": { 05:58:50 DEBUG { 05:58:50 DEBUG 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:58:50 DEBUG { 05:58:50 DEBUG { 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "metric": { 05:58:50 DEBUG { 05:58:50 DEBUG "resultType": "vector", 05:58:50 DEBUG "value": [ 05:58:50 DEBUG }, 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "instance": "10.142.0.107:9100" 05:58:50 DEBUG "pod": "ds-idrepo-0" 05:58:50 DEBUG "metric": { 05:58:50 DEBUG 05:58:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:58:50 DEBUG "result": [ 05:58:50 DEBUG }, 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "metric": { 05:58:50 DEBUG 05:58:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:58:50 DEBUG "metric": { 05:58:50 DEBUG "result": [ 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "instance": "10.142.0.107:9100" 05:58:50 DEBUG "pod": "ds-idrepo-0" 05:58:50 DEBUG }, 05:58:50 DEBUG }, 05:58:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG { 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:50 DEBUG "pod": "ds-cts-0" 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG "pod": "ds-idrepo-0" 05:58:50 DEBUG { 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG }, 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "value": [ 05:58:50 DEBUG }, 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG "metric": { 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG }, 05:58:50 DEBUG "metric": { 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "value": [ 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "pod": "ds-cts-0" 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:50 DEBUG [...] 05:58:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "pod": "ds-cts-2" 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG [...] 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 05:58:50 DEBUG }, 05:58:50 DEBUG "instance": "10.142.0.43:9100" 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "pod": "ds-idrepo-2" 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG "value": [ 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG "pod": "ds-cts-2" 05:58:50 DEBUG }, 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG 05:58:50 DEBUG [...] 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:50 DEBUG 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "instance": "10.142.0.43:9100" 05:58:50 DEBUG }, 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG "2663.0666666666666" 05:58:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:50 DEBUG 05:58:50 DEBUG "480589.5" 05:58:50 DEBUG }, 05:58:50 DEBUG "pod": "ds-cts-2" 05:58:50 DEBUG "1669.6786466471908" 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG }, 05:58:50 DEBUG "value": [ 05:58:50 DEBUG "0" 05:58:50 DEBUG 05:58:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:58:50 DEBUG ] 05:58:50 DEBUG }, 05:58:50 DEBUG 05:58:50 DEBUG ] 05:58:50 DEBUG "value": [ 05:58:50 DEBUG }, 05:58:50 DEBUG ] 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG "pod": "ds-idrepo-2" 05:58:50 DEBUG "0" 05:58:50 DEBUG "pod": "ds-cts-2" 05:58:50 DEBUG "0.2501999999999725" 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG ] 05:58:50 DEBUG "instance": "10.142.0.43:9100" 05:58:50 DEBUG }, 05:58:50 DEBUG } 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 05:58:50 DEBUG } 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG "value": [ 05:58:50 DEBUG } 05:58:50 DEBUG "8.04467881047252" 05:58:50 DEBUG }, 05:58:50 DEBUG ] 05:58:50 DEBUG }, 05:58:50 DEBUG ] 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG "0" 05:58:50 DEBUG } 05:58:50 DEBUG }, 05:58:50 DEBUG "value": [ 05:58:50 DEBUG ] 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:58:50 DEBUG ] 05:58:50 DEBUG "0" 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG ] 05:58:50 DEBUG ] 05:58:50 DEBUG "value": [ 05:58:50 DEBUG } 05:58:50 DEBUG "value": [ 05:58:50 DEBUG } 05:58:50 DEBUG "81087691.96666667" 05:58:50 DEBUG ] 05:58:50 DEBUG ] 05:58:50 DEBUG "value": [ 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG } 05:58:50 DEBUG "2667.866666666667" 05:58:50 DEBUG }, 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG "23.3" 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG ] 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG ] 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG "7831937024" 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG "value": [ 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG "0" 05:58:50 DEBUG } 05:58:50 DEBUG "0.0005" 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG "3127176115200" 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG 1713589130, 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG "0.092" 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG ] 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:58:50 DEBUG } 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589187" 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589187" 05:59:47 ERROR [http_cmd]: ERROR 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 ERROR --- status code --- 05:59:47 DEBUG --- status code --- 05:59:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 ERROR --- http response --- 05:59:47 DEBUG --- http response --- 05:59:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 05:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 05:59:47 DEBUG ----- output ----- 05:59:47 DEBUG { 05:59:47 DEBUG "status": "success", 05:59:47 DEBUG "data": { 05:59:47 DEBUG "resultType": "vector", 05:59:47 DEBUG "result": [ 05:59:47 DEBUG { 05:59:47 DEBUG "metric": { 05:59:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG [...] 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 1713589187, 05:59:47 DEBUG "2678.7666666666664" 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG } 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589187" 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 DEBUG --- status code --- 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 DEBUG --- http response --- 05:59:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589187" 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 DEBUG --- status code --- 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 DEBUG --- http response --- 05:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:59:47 DEBUG ----- output ----- 05:59:47 DEBUG { 05:59:47 DEBUG "status": "success", 05:59:47 DEBUG "data": { 05:59:47 DEBUG "resultType": "vector", 05:59:47 DEBUG "result": [ 05:59:47 DEBUG { 05:59:47 DEBUG "metric": { 05:59:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG [...] 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG "pod": "idm-b85f9d978-wnhxb" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 1713589187, 05:59:47 DEBUG "33102283.504926488" 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG } 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589187" 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589187" 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 DEBUG --- status code --- 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 DEBUG --- http response --- 05:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:47 DEBUG ----- output ----- 05:59:47 DEBUG { 05:59:47 DEBUG "status": "success", 05:59:47 DEBUG "data": { 05:59:47 DEBUG "resultType": "vector", 05:59:47 DEBUG "result": [ 05:59:47 DEBUG { 05:59:47 DEBUG "metric": { 05:59:47 DEBUG "pod": "am-767f6b67b5-9zw22" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG [...] 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 1713589187, 05:59:47 DEBUG "0" 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG } 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589187" 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 DEBUG --- status code --- 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 DEBUG --- http response --- 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589187" 05:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:47 DEBUG ----- output ----- 05:59:47 DEBUG { 05:59:47 DEBUG "status": "success", 05:59:47 DEBUG "data": { 05:59:47 DEBUG "resultType": "vector", 05:59:47 DEBUG "result": [ 05:59:47 DEBUG { 05:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589187" 05:59:47 DEBUG "metric": { 05:59:47 DEBUG "instance": "10.142.0.107:9100" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG [...] 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG "instance": "10.142.0.43:9100" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 1713589187, 05:59:47 DEBUG "0.06333333333334015" 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG } 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 DEBUG --- status code --- 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 DEBUG --- http response --- 05:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 05:59:47 DEBUG ----- output ----- 05:59:47 DEBUG { 05:59:47 DEBUG "status": "success", 05:59:47 DEBUG "data": { 05:59:47 DEBUG "resultType": "vector", 05:59:47 DEBUG "result": [ 05:59:47 DEBUG { 05:59:47 DEBUG "metric": { 05:59:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG [...] 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 1713589187, 05:59:47 DEBUG "0" 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG } 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 DEBUG --- status code --- 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 DEBUG --- http response --- 05:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:47 DEBUG ----- output ----- 05:59:47 DEBUG { 05:59:47 DEBUG "status": "success", 05:59:47 DEBUG "data": { 05:59:47 DEBUG "resultType": "vector", 05:59:47 DEBUG "result": [ 05:59:47 DEBUG { 05:59:47 DEBUG "metric": { 05:59:47 DEBUG "pod": "ds-idrepo-0" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG [...] 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG "pod": "ds-idrepo-2" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 1713589187, 05:59:47 DEBUG "0" 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG } 05:59:47 INFO [http_cmd]: http status code OK 05:59:47 DEBUG --- status code --- 05:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:47 DEBUG --- http response --- 05:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:47 DEBUG ----- output ----- 05:59:47 DEBUG { 05:59:47 DEBUG "status": "success", 05:59:47 DEBUG "data": { 05:59:47 DEBUG "resultType": "vector", 05:59:47 DEBUG "result": [ 05:59:47 DEBUG { 05:59:47 DEBUG "metric": { 05:59:47 DEBUG "instance": "10.142.0.107:9100" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG [...] 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG 05:59:47 DEBUG "instance": "10.142.0.43:9100" 05:59:47 DEBUG }, 05:59:47 DEBUG "value": [ 05:59:47 DEBUG 1713589187, 05:59:47 DEBUG "80591127.5" 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG ] 05:59:47 DEBUG } 05:59:47 DEBUG } 05:59:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589189" 05:59:49 INFO [http_cmd]: http status code OK 05:59:49 DEBUG --- status code --- 05:59:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:49 DEBUG --- http response --- 05:59:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:49 DEBUG ----- output ----- 05:59:49 DEBUG { 05:59:49 DEBUG "status": "success", 05:59:49 DEBUG "data": { 05:59:49 DEBUG "resultType": "vector", 05:59:49 DEBUG "result": [ 05:59:49 DEBUG { 05:59:49 DEBUG "metric": { 05:59:49 DEBUG "pod": "am-767f6b67b5-9zw22" 05:59:49 DEBUG }, 05:59:49 DEBUG "value": [ 05:59:49 DEBUG 05:59:49 DEBUG 05:59:49 DEBUG 05:59:49 DEBUG [...] 05:59:49 DEBUG 05:59:49 DEBUG 05:59:49 DEBUG 05:59:49 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:49 DEBUG }, 05:59:49 DEBUG "value": [ 05:59:49 DEBUG 1713589189, 05:59:49 DEBUG "2659.3" 05:59:49 DEBUG ] 05:59:49 DEBUG } 05:59:49 DEBUG ] 05:59:49 DEBUG } 05:59:49 DEBUG } 05:59:50 INFO 05:59:50 INFO [loop_until]: kubectl --namespace=xlou top pods 05:59:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:59:50 INFO 05:59:50 INFO [loop_until]: kubectl --namespace=xlou top node 05:59:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589190" 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 DEBUG --- status code --- 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 DEBUG --- status code --- 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [loop_until]: OK (rc = 0) 05:59:50 INFO [loop_until]: OK (rc = 0) 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 DEBUG --- status code --- 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 DEBUG --- status code --- 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- status code --- 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 DEBUG --- stdout --- 05:59:50 DEBUG --- stdout --- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG --- status code --- 05:59:50 INFO [http_cmd]: http status code OK 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 16899m 8653Mi am-767f6b67b5-qcqz2 17267m 8711Mi am-767f6b67b5-vbltz 18207m 8652Mi ds-cts-0 18235m 10627Mi ds-cts-1 17590m 10723Mi ds-cts-2 18445m 10611Mi ds-idrepo-0 7585m 3714Mi ds-idrepo-1 7580m 3705Mi ds-idrepo-2 7625m 3627Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8105m 7377Mi 05:59:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 6207Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7983m 26% 7377Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17206m 57% 11367Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3374Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18783m 62% 13775Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5197Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17434m 58% 11593Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19059m 63% 3005Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19334m 64% 10865Mi 9% gke-xlou-cdm-ds-0f601751-j488 8230m 27% 6938Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17509m 58% 15722Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 7778m 26% 7596Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19145m 64% 13215Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8177m 51% 11183Mi 19% 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- status code --- 05:59:50 DEBUG { 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG --- stderr --- 05:59:50 DEBUG --- stderr --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG --- http response --- 05:59:50 DEBUG "data": { 05:59:50 DEBUG { 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG { 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 05:59:50 DEBUG "data": { 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "result": [ 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG "data": { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG "data": { 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "result": [ 05:59:50 DEBUG ----- output ----- 05:59:50 DEBUG { 05:59:50 DEBUG "data": { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "data": { 05:59:50 DEBUG "data": { 05:59:50 DEBUG "data": { 05:59:50 DEBUG { 05:59:50 DEBUG "data": { 05:59:50 DEBUG "data": { 05:59:50 DEBUG { 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "data": { 05:59:50 DEBUG "data": { 05:59:50 DEBUG "data": { 05:59:50 DEBUG "data": { 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "status": "success", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "data": { 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:59:50 DEBUG "data": { 05:59:50 DEBUG { 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "data": { 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG }, 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG "resultType": "vector", 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG }, 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "instance": "10.142.0.107:9100" 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "result": [ 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG "pod": "ds-cts-0" 05:59:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 05:59:50 DEBUG 05:59:50 DEBUG { 05:59:50 DEBUG }, 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "pod": "ds-cts-0" 05:59:50 DEBUG "instance": "10.142.0.107:9100" 05:59:50 DEBUG "instance": "10.142.0.107:9100" 05:59:50 DEBUG { 05:59:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:59:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:59:50 DEBUG { 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG 05:59:50 DEBUG "metric": { 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "pod": "ds-idrepo-0" 05:59:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:59:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 05:59:50 DEBUG "pod": "ds-idrepo-0" 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG "metric": { 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG "metric": { 05:59:50 DEBUG 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG "pod": "ds-idrepo-0" 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "pod": "ds-cts-0" 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "pod": "am-767f6b67b5-9zw22" 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG }, 05:59:50 DEBUG 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG [...] 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG "instance": "10.142.0.43:9100" 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "instance": "10.142.0.43:9100" 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "8.037765122802925" 05:59:50 DEBUG 05:59:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG "pod": "ds-idrepo-2" 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG "pod": "ds-cts-2" 05:59:50 DEBUG }, 05:59:50 DEBUG 05:59:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:59:50 DEBUG 05:59:50 DEBUG 05:59:50 DEBUG ] 05:59:50 DEBUG "pod": "ds-cts-2" 05:59:50 DEBUG }, 05:59:50 DEBUG "480400" 05:59:50 DEBUG 05:59:50 DEBUG "value": [ 05:59:50 DEBUG }, 05:59:50 DEBUG 05:59:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 05:59:50 DEBUG "pod": "ds-idrepo-2" 05:59:50 DEBUG }, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG "instance": "10.142.0.43:9100" 05:59:50 DEBUG 05:59:50 DEBUG } 05:59:50 DEBUG }, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG ] 05:59:50 DEBUG "pod": "ds-cts-2" 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:50 DEBUG }, 05:59:50 DEBUG }, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "pod": "ds-cts-2" 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG }, 05:59:50 DEBUG "pod": "am-767f6b67b5-vbltz" 05:59:50 DEBUG ] 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG "0.16346666666666748" 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG }, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "30" 05:59:50 DEBUG }, 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG }, 05:59:50 DEBUG } 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "7831986176" 05:59:50 DEBUG }, 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG "0" 05:59:50 DEBUG "value": [ 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "80587115.23333333" 05:59:50 DEBUG ] 05:59:50 DEBUG "value": [ 05:59:50 DEBUG "0.092" 05:59:50 DEBUG "0" 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "value": [ 05:59:50 DEBUG } 05:59:50 DEBUG "0" 05:59:50 DEBUG ] 05:59:50 DEBUG "value": [ 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG "0" 05:59:50 DEBUG "0" 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG ] 05:59:50 DEBUG ] 05:59:50 DEBUG "3131532270592" 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG 1713589190, 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG "2659.7" 05:59:50 DEBUG ] 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG "0.0005154888888888889" 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG "2659.1" 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG "0" 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG ] 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 05:59:50 DEBUG } 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589247" 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589247" 06:00:47 ERROR [http_cmd]: ERROR 06:00:47 ERROR --- status code --- 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:00:47 DEBUG --- status code --- 06:00:47 ERROR --- http response --- 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:00:47 DEBUG --- http response --- 06:00:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:00:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:47 DEBUG ----- output ----- 06:00:47 DEBUG { 06:00:47 DEBUG "status": "success", 06:00:47 DEBUG "data": { 06:00:47 DEBUG "resultType": "vector", 06:00:47 DEBUG "result": [ 06:00:47 DEBUG { 06:00:47 DEBUG "metric": { 06:00:47 DEBUG "pod": "am-767f6b67b5-9zw22" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG [...] 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 1713589247, 06:00:47 DEBUG "2664.6" 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG } 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589247" 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 DEBUG --- status code --- 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG --- http response --- 06:00:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589247" 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 DEBUG --- status code --- 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG --- http response --- 06:00:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 06:00:47 DEBUG ----- output ----- 06:00:47 DEBUG { 06:00:47 DEBUG "status": "success", 06:00:47 DEBUG "data": { 06:00:47 DEBUG "resultType": "vector", 06:00:47 DEBUG "result": [ 06:00:47 DEBUG { 06:00:47 DEBUG "metric": { 06:00:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG [...] 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 1713589247, 06:00:47 DEBUG "9468.723044543352" 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG } 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589247" 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 DEBUG --- status code --- 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG --- http response --- 06:00:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:47 DEBUG ----- output ----- 06:00:47 DEBUG { 06:00:47 DEBUG "status": "success", 06:00:47 DEBUG "data": { 06:00:47 DEBUG "resultType": "vector", 06:00:47 DEBUG "result": [ 06:00:47 DEBUG { 06:00:47 DEBUG "metric": { 06:00:47 DEBUG "pod": "am-767f6b67b5-9zw22" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG [...] 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 1713589247, 06:00:47 DEBUG "0" 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG } 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589247" 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589247" 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589247" 06:00:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589247" 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 DEBUG --- status code --- 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG --- http response --- 06:00:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:00:47 DEBUG ----- output ----- 06:00:47 DEBUG { 06:00:47 DEBUG "status": "success", 06:00:47 DEBUG "data": { 06:00:47 DEBUG "resultType": "vector", 06:00:47 DEBUG "result": [ 06:00:47 DEBUG { 06:00:47 DEBUG "metric": { 06:00:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG [...] 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 1713589247, 06:00:47 DEBUG "18.70588624366155" 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG } 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 DEBUG --- status code --- 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG --- http response --- 06:00:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:47 DEBUG ----- output ----- 06:00:47 DEBUG { 06:00:47 DEBUG "status": "success", 06:00:47 DEBUG "data": { 06:00:47 DEBUG "resultType": "vector", 06:00:47 DEBUG "result": [ 06:00:47 DEBUG { 06:00:47 DEBUG "metric": { 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 DEBUG "instance": "10.142.0.107:9100" 06:00:47 INFO [http_cmd]: http status code OK 06:00:47 DEBUG --- status code --- 06:00:47 DEBUG }, 06:00:47 DEBUG --- status code --- 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG "value": [ 06:00:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:47 DEBUG --- http response --- 06:00:47 DEBUG 06:00:47 DEBUG --- http response --- 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG [...] 06:00:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:47 DEBUG 06:00:47 DEBUG ----- output ----- 06:00:47 DEBUG ----- output ----- 06:00:47 DEBUG 06:00:47 DEBUG { 06:00:47 DEBUG { 06:00:47 DEBUG 06:00:47 DEBUG "status": "success", 06:00:47 DEBUG "status": "success", 06:00:47 DEBUG "instance": "10.142.0.43:9100" 06:00:47 DEBUG "data": { 06:00:47 DEBUG "data": { 06:00:47 DEBUG }, 06:00:47 DEBUG "resultType": "vector", 06:00:47 DEBUG "resultType": "vector", 06:00:47 DEBUG "value": [ 06:00:47 DEBUG "result": [ 06:00:47 DEBUG "result": [ 06:00:47 DEBUG { 06:00:47 DEBUG 1713589247, 06:00:47 DEBUG { 06:00:47 DEBUG "metric": { 06:00:47 DEBUG "0.06466666666667416" 06:00:47 DEBUG "metric": { 06:00:47 DEBUG "instance": "10.142.0.107:9100" 06:00:47 DEBUG ] 06:00:47 DEBUG "pod": "ds-idrepo-0" 06:00:47 DEBUG }, 06:00:47 DEBUG } 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG ] 06:00:47 DEBUG "value": [ 06:00:47 DEBUG 06:00:47 DEBUG } 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG } 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG [...] 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG 06:00:47 DEBUG "instance": "10.142.0.43:9100" 06:00:47 DEBUG }, 06:00:47 DEBUG "value": [ 06:00:47 DEBUG [...] 06:00:47 DEBUG 1713589247, 06:00:47 DEBUG 06:00:47 DEBUG "80685283.73333333" 06:00:47 DEBUG 06:00:47 DEBUG ] 06:00:47 DEBUG 06:00:47 DEBUG } 06:00:47 DEBUG "pod": "ds-idrepo-2" 06:00:47 DEBUG ] 06:00:47 DEBUG }, 06:00:47 DEBUG } 06:00:47 DEBUG "value": [ 06:00:47 DEBUG } 06:00:47 DEBUG 1713589247, 06:00:47 DEBUG "0" 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG ] 06:00:47 DEBUG } 06:00:47 DEBUG } 06:00:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589249" 06:00:49 INFO [http_cmd]: http status code OK 06:00:49 DEBUG --- status code --- 06:00:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:49 DEBUG --- http response --- 06:00:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:49 DEBUG ----- output ----- 06:00:49 DEBUG { 06:00:49 DEBUG "status": "success", 06:00:49 DEBUG "data": { 06:00:49 DEBUG "resultType": "vector", 06:00:49 DEBUG "result": [ 06:00:49 DEBUG { 06:00:49 DEBUG "metric": { 06:00:49 DEBUG "pod": "am-767f6b67b5-9zw22" 06:00:49 DEBUG }, 06:00:49 DEBUG "value": [ 06:00:49 DEBUG 06:00:49 DEBUG 06:00:49 DEBUG 06:00:49 DEBUG [...] 06:00:49 DEBUG 06:00:49 DEBUG 06:00:49 DEBUG 06:00:49 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:49 DEBUG }, 06:00:49 DEBUG "value": [ 06:00:49 DEBUG 1713589249, 06:00:49 DEBUG "2673.3333333333335" 06:00:49 DEBUG ] 06:00:49 DEBUG } 06:00:49 DEBUG ] 06:00:49 DEBUG } 06:00:49 DEBUG } 06:00:50 INFO 06:00:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:00:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:00:50 INFO 06:00:50 INFO [loop_until]: kubectl --namespace=xlou top node 06:00:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589250" 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [loop_until]: OK (rc = 0) 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 DEBUG --- status code --- 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 INFO [loop_until]: OK (rc = 0) 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- stdout --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- stdout --- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17385m 8649Mi am-767f6b67b5-qcqz2 17459m 8708Mi am-767f6b67b5-vbltz 18882m 8651Mi ds-cts-0 18376m 10628Mi ds-cts-1 17023m 10727Mi ds-cts-2 19497m 10577Mi ds-idrepo-0 7544m 3714Mi ds-idrepo-1 7653m 3705Mi ds-idrepo-2 8022m 3626Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 16m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8050m 7378Mi 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:50 DEBUG --- http response --- 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6208Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7995m 26% 7384Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17230m 57% 11362Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3375Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18585m 62% 13776Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 608m 2% 5077Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17442m 58% 11609Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19234m 64% 3008Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19520m 65% 10858Mi 9% gke-xlou-cdm-ds-0f601751-j488 8253m 27% 6944Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17316m 58% 15726Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8060m 26% 7590Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19506m 65% 13209Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8184m 51% 11186Mi 19% 06:00:50 DEBUG --- http response --- 06:00:50 INFO [http_cmd]: http status code OK 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG --- stderr --- 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG --- stderr --- 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:50 DEBUG --- status code --- 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:50 DEBUG 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG { 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG 06:00:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "data": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG { 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG { 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG { 06:00:50 DEBUG --- http response --- 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG "result": [ 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:00:50 DEBUG "data": { 06:00:50 DEBUG { 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "data": { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG { 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "data": { 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG "data": { 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "data": { 06:00:50 DEBUG "data": { 06:00:50 DEBUG ----- output ----- 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "data": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG { 06:00:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "data": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "data": { 06:00:50 DEBUG "data": { 06:00:50 DEBUG "data": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG { 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "data": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG }, 06:00:50 DEBUG "data": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "status": "success", 06:00:50 DEBUG { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG { 06:00:50 DEBUG "data": { 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG { 06:00:50 DEBUG "data": { 06:00:50 DEBUG { 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "result": [ 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG "data": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG 06:00:50 DEBUG "result": [ 06:00:50 DEBUG { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "resultType": "vector", 06:00:50 DEBUG "result": [ 06:00:50 DEBUG "metric": { 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "result": [ 06:00:50 DEBUG { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "pod": "ds-cts-0" 06:00:50 DEBUG "result": [ 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:00:50 DEBUG }, 06:00:50 DEBUG { 06:00:50 DEBUG "instance": "10.142.0.107:9100" 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:00:50 DEBUG "pod": "ds-idrepo-0" 06:00:50 DEBUG { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:00:50 DEBUG "pod": "ds-cts-0" 06:00:50 DEBUG }, 06:00:50 DEBUG { 06:00:50 DEBUG [...] 06:00:50 DEBUG "instance": "10.142.0.107:9100" 06:00:50 DEBUG { 06:00:50 DEBUG { 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "metric": { 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "pod": "ds-idrepo-0" 06:00:50 DEBUG }, 06:00:50 DEBUG }, 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "pod": "ds-cts-0" 06:00:50 DEBUG }, 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "metric": { 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG "metric": { 06:00:50 DEBUG "metric": { 06:00:50 DEBUG 06:00:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:00:50 DEBUG 06:00:50 DEBUG "pod": "ds-idrepo-0" 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 06:00:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:00:50 DEBUG "instance": "10.142.0.107:9100" 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG }, 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "value": [ 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG "2673.766666666667" 06:00:50 DEBUG 06:00:50 DEBUG [...] 06:00:50 DEBUG [...] 06:00:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "pod": "ds-cts-2" 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "pod": "ds-cts-2" 06:00:50 DEBUG 06:00:50 DEBUG ] 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG "instance": "10.142.0.43:9100" 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG }, 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:00:50 DEBUG "pod": "ds-cts-2" 06:00:50 DEBUG }, 06:00:50 DEBUG 06:00:50 DEBUG } 06:00:50 DEBUG "instance": "10.142.0.43:9100" 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG "value": [ 06:00:50 DEBUG }, 06:00:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:50 DEBUG 06:00:50 DEBUG "pod": "ds-idrepo-2" 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:50 DEBUG 06:00:50 DEBUG "pod": "ds-cts-2" 06:00:50 DEBUG }, 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 06:00:50 DEBUG ] 06:00:50 DEBUG }, 06:00:50 DEBUG 06:00:50 DEBUG 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG }, 06:00:50 DEBUG "pod": "ds-idrepo-2" 06:00:50 DEBUG }, 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG }, 06:00:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:00:50 DEBUG } 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:00:50 DEBUG "instance": "10.142.0.43:9100" 06:00:50 DEBUG "479721" 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "0" 06:00:50 DEBUG "value": [ 06:00:50 DEBUG }, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "0" 06:00:50 DEBUG }, 06:00:50 DEBUG } 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG }, 06:00:50 DEBUG }, 06:00:50 DEBUG ] 06:00:50 DEBUG "3134948566016" 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG ] 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "value": [ 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "7832195072" 06:00:50 DEBUG "34.63333333333333" 06:00:50 DEBUG ] 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "80678277.6" 06:00:50 DEBUG "value": [ 06:00:50 DEBUG "value": [ 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG "2673.6666666666665" 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "0" 06:00:50 DEBUG } 06:00:50 DEBUG "565.2968981816927" 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG "0.00016666666666666666" 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG ] 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG 1713589250, 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG "0" 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG "0" 06:00:50 DEBUG } 06:00:50 DEBUG "8.033603355862885" 06:00:50 DEBUG "0.17213333333332723" 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG "0.092" 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG ] 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:00:50 DEBUG } 06:01:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589307" 06:01:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589307" 06:01:47 ERROR [http_cmd]: ERROR 06:01:47 ERROR --- status code --- 06:01:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:01:47 ERROR --- http response --- 06:01:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:01:47 INFO [http_cmd]: http status code OK 06:01:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:01:47 DEBUG --- status code --- 06:01:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:47 DEBUG --- http response --- 06:01:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:47 DEBUG ----- output ----- 06:01:47 DEBUG { 06:01:47 DEBUG "status": "success", 06:01:47 DEBUG "data": { 06:01:47 DEBUG "resultType": "vector", 06:01:47 DEBUG "result": [ 06:01:47 DEBUG { 06:01:47 DEBUG "metric": { 06:01:47 DEBUG "pod": "am-767f6b67b5-9zw22" 06:01:47 DEBUG }, 06:01:47 DEBUG "value": [ 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG [...] 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:47 DEBUG }, 06:01:47 DEBUG "value": [ 06:01:47 DEBUG 1713589307, 06:01:47 DEBUG "2668.233333333333" 06:01:47 DEBUG ] 06:01:47 DEBUG } 06:01:47 DEBUG ] 06:01:47 DEBUG } 06:01:47 DEBUG } 06:01:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589307" 06:01:47 INFO [http_cmd]: http status code OK 06:01:47 DEBUG --- status code --- 06:01:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:47 DEBUG --- http response --- 06:01:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:01:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589307" 06:01:47 INFO [http_cmd]: http status code OK 06:01:47 DEBUG --- status code --- 06:01:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:47 DEBUG --- http response --- 06:01:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:01:47 DEBUG ----- output ----- 06:01:47 DEBUG { 06:01:47 DEBUG "status": "success", 06:01:47 DEBUG "data": { 06:01:47 DEBUG "resultType": "vector", 06:01:47 DEBUG "result": [ 06:01:47 DEBUG { 06:01:47 DEBUG "metric": { 06:01:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:01:47 DEBUG }, 06:01:47 DEBUG "value": [ 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG [...] 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG 06:01:47 DEBUG "pod": "ds-cts-2" 06:01:47 DEBUG }, 06:01:47 DEBUG "value": [ 06:01:47 DEBUG 1713589307, 06:01:47 DEBUG "630468.8236407975" 06:01:47 DEBUG ] 06:01:47 DEBUG } 06:01:47 DEBUG ] 06:01:47 DEBUG } 06:01:47 DEBUG } 06:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589308" 06:01:48 INFO [http_cmd]: http status code OK 06:01:48 DEBUG --- status code --- 06:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:48 DEBUG --- http response --- 06:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:48 DEBUG ----- output ----- 06:01:48 DEBUG { 06:01:48 DEBUG "status": "success", 06:01:48 DEBUG "data": { 06:01:48 DEBUG "resultType": "vector", 06:01:48 DEBUG "result": [ 06:01:48 DEBUG { 06:01:48 DEBUG "metric": { 06:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:01:48 DEBUG }, 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG [...] 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:48 DEBUG }, 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 1713589308, 06:01:48 DEBUG "0" 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG } 06:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589308" 06:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589308" 06:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589308" 06:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589308" 06:01:48 INFO [http_cmd]: http status code OK 06:01:48 DEBUG --- status code --- 06:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:48 DEBUG --- http response --- 06:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:01:48 DEBUG ----- output ----- 06:01:48 DEBUG { 06:01:48 DEBUG "status": "success", 06:01:48 DEBUG "data": { 06:01:48 DEBUG "resultType": "vector", 06:01:48 DEBUG "result": [ 06:01:48 DEBUG { 06:01:48 DEBUG "metric": { 06:01:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:01:48 DEBUG }, 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG [...] 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:48 DEBUG }, 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 1713589308, 06:01:48 DEBUG "18.860550233646347" 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG } 06:01:48 INFO [http_cmd]: http status code OK 06:01:48 DEBUG --- status code --- 06:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:48 DEBUG --- http response --- 06:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:48 DEBUG ----- output ----- 06:01:48 DEBUG { 06:01:48 DEBUG "status": "success", 06:01:48 DEBUG "data": { 06:01:48 DEBUG "resultType": "vector", 06:01:48 DEBUG "result": [ 06:01:48 DEBUG { 06:01:48 DEBUG "metric": { 06:01:48 DEBUG "instance": "10.142.0.107:9100" 06:01:48 DEBUG }, 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG [...] 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG "instance": "10.142.0.43:9100" 06:01:48 DEBUG }, 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 1713589308, 06:01:48 DEBUG "81072549.16666666" 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG } 06:01:48 INFO [http_cmd]: http status code OK 06:01:48 DEBUG --- status code --- 06:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:48 DEBUG --- http response --- 06:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:48 DEBUG ----- output ----- 06:01:48 INFO [http_cmd]: http status code OK 06:01:48 DEBUG { 06:01:48 DEBUG --- status code --- 06:01:48 DEBUG "status": "success", 06:01:48 DEBUG "data": { 06:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:48 DEBUG "resultType": "vector", 06:01:48 DEBUG --- http response --- 06:01:48 DEBUG "result": [ 06:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:48 DEBUG { 06:01:48 DEBUG ----- output ----- 06:01:48 DEBUG "metric": { 06:01:48 DEBUG { 06:01:48 DEBUG "instance": "10.142.0.107:9100" 06:01:48 DEBUG "status": "success", 06:01:48 DEBUG }, 06:01:48 DEBUG "data": { 06:01:48 DEBUG "value": [ 06:01:48 DEBUG "resultType": "vector", 06:01:48 DEBUG 06:01:48 DEBUG "result": [ 06:01:48 DEBUG 06:01:48 DEBUG { 06:01:48 DEBUG 06:01:48 DEBUG "metric": { 06:01:48 DEBUG [...] 06:01:48 DEBUG "pod": "ds-idrepo-0" 06:01:48 DEBUG 06:01:48 DEBUG }, 06:01:48 DEBUG 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 06:01:48 DEBUG "instance": "10.142.0.43:9100" 06:01:48 DEBUG 06:01:48 DEBUG 06:01:48 DEBUG }, 06:01:48 DEBUG 06:01:48 DEBUG "value": [ 06:01:48 DEBUG [...] 06:01:48 DEBUG 1713589308, 06:01:48 DEBUG 06:01:48 DEBUG "0.07466666666667077" 06:01:48 DEBUG 06:01:48 DEBUG ] 06:01:48 DEBUG 06:01:48 DEBUG } 06:01:48 DEBUG "pod": "ds-idrepo-2" 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG } 06:01:48 DEBUG }, 06:01:48 DEBUG "value": [ 06:01:48 DEBUG 1713589308, 06:01:48 DEBUG "0" 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG ] 06:01:48 DEBUG } 06:01:48 DEBUG } 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "result": [ 06:01:50 DEBUG { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG "2674.3333333333335" 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 INFO 06:01:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:01:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:01:50 INFO 06:01:50 INFO [loop_until]: kubectl --namespace=xlou top node 06:01:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589310" 06:01:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589310" 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [loop_until]: OK (rc = 0) 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [loop_until]: OK (rc = 0) 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 DEBUG --- status code --- 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- stdout --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- stdout --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG --- http response --- 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 DEBUG --- status code --- 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6209Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8008m 26% 7385Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17287m 57% 11360Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3370Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18589m 62% 13775Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5143Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17463m 58% 11604Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19184m 64% 3010Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19537m 65% 10863Mi 9% gke-xlou-cdm-ds-0f601751-j488 8276m 27% 6942Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17499m 58% 15720Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8046m 26% 7602Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19046m 63% 13213Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8177m 51% 11184Mi 19% 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17853m 8652Mi am-767f6b67b5-qcqz2 17360m 8714Mi am-767f6b67b5-vbltz 18868m 8651Mi ds-cts-0 18811m 10628Mi ds-cts-1 17718m 10724Mi ds-cts-2 18992m 10602Mi ds-idrepo-0 7741m 3714Mi ds-idrepo-1 7686m 3706Mi ds-idrepo-2 7884m 3626Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 16m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7988m 7379Mi 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG --- http response --- 06:01:50 INFO [http_cmd]: http status code OK 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG --- status code --- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:01:50 DEBUG --- stderr --- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG --- stderr --- 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:50 DEBUG { 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:50 DEBUG { 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:01:50 DEBUG 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG { 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG --- http response --- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG { 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG { 06:01:50 DEBUG { 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG { 06:01:50 DEBUG { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG { 06:01:50 DEBUG { 06:01:50 DEBUG { 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG { 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG { 06:01:50 DEBUG { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "data": { 06:01:50 DEBUG { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "data": { 06:01:50 DEBUG ----- output ----- 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "data": { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "data": { 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "status": "success", 06:01:50 DEBUG { 06:01:50 DEBUG { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG { 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "metric": { 06:01:50 DEBUG { 06:01:50 DEBUG { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG "metric": { 06:01:50 DEBUG { 06:01:50 DEBUG "data": { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "result": [ 06:01:50 DEBUG { 06:01:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG { 06:01:50 DEBUG "pod": "ds-cts-0" 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "resultType": "vector", 06:01:50 DEBUG "pod": "ds-cts-0" 06:01:50 DEBUG "instance": "10.142.0.107:9100" 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "pod": "ds-idrepo-0" 06:01:50 DEBUG { 06:01:50 DEBUG "metric": { 06:01:50 DEBUG }, 06:01:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:01:50 DEBUG "pod": "ds-idrepo-0" 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "pod": "ds-cts-0" 06:01:50 DEBUG "metric": { 06:01:50 DEBUG }, 06:01:50 DEBUG "pod": "ds-idrepo-0" 06:01:50 DEBUG "result": [ 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG "instance": "10.142.0.107:9100" 06:01:50 DEBUG }, 06:01:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:01:50 DEBUG }, 06:01:50 DEBUG "metric": { 06:01:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG "instance": "10.142.0.107:9100" 06:01:50 DEBUG }, 06:01:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG { 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:01:50 DEBUG }, 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "metric": { 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG }, 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG [...] 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG [...] 06:01:50 DEBUG [...] 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG 06:01:50 DEBUG "pod": "ds-cts-2" 06:01:50 DEBUG 06:01:50 DEBUG [...] 06:01:50 DEBUG "pod": "ds-cts-2" 06:01:50 DEBUG "instance": "10.142.0.43:9100" 06:01:50 DEBUG 06:01:50 DEBUG }, 06:01:50 DEBUG 06:01:50 DEBUG "pod": "ds-cts-2" 06:01:50 DEBUG 06:01:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:01:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:50 DEBUG "pod": "ds-idrepo-2" 06:01:50 DEBUG 06:01:50 DEBUG "pod": "ds-cts-2" 06:01:50 DEBUG "instance": "10.142.0.43:9100" 06:01:50 DEBUG }, 06:01:50 DEBUG "pod": "ds-idrepo-2" 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG "instance": "10.142.0.43:9100" 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG }, 06:01:50 DEBUG 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "value": [ 06:01:50 DEBUG }, 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG "3103.8140944683" 06:01:50 DEBUG ] 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG 06:01:50 DEBUG "value": [ 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:50 DEBUG "value": [ 06:01:50 DEBUG } 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG }, 06:01:50 DEBUG "29.966666666666665" 06:01:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG "0" 06:01:50 DEBUG 06:01:50 DEBUG "0" 06:01:50 DEBUG "0.0008666666666666667" 06:01:50 DEBUG "81066272.1" 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG }, 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG ] 06:01:50 DEBUG "0" 06:01:50 DEBUG "7834157056" 06:01:50 DEBUG "2671.7" 06:01:50 DEBUG "0" 06:01:50 DEBUG "value": [ 06:01:50 DEBUG ] 06:01:50 DEBUG }, 06:01:50 DEBUG "3139101479936" 06:01:50 DEBUG ] 06:01:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG "0.21929999999993582" 06:01:50 DEBUG "value": [ 06:01:50 DEBUG "2674.3333333333335" 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG } 06:01:50 DEBUG "value": [ 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG }, 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG "8.039093594826126" 06:01:50 DEBUG ] 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG "value": [ 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG "479868.5" 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG "0.092" 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG 1713589310, 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG "0" 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG ] 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:01:50 DEBUG } 06:02:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589367" 06:02:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589367" 06:02:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589367" 06:02:47 ERROR [http_cmd]: ERROR 06:02:47 ERROR --- status code --- 06:02:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:02:47 ERROR --- http response --- 06:02:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:02:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:02:47 INFO [http_cmd]: http status code OK 06:02:47 DEBUG --- status code --- 06:02:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:47 DEBUG --- http response --- 06:02:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:47 DEBUG ----- output ----- 06:02:47 DEBUG { 06:02:47 DEBUG "status": "success", 06:02:47 DEBUG "data": { 06:02:47 DEBUG "resultType": "vector", 06:02:47 DEBUG "result": [ 06:02:47 DEBUG { 06:02:47 DEBUG "metric": { 06:02:47 DEBUG "pod": "am-767f6b67b5-9zw22" 06:02:47 DEBUG }, 06:02:47 DEBUG "value": [ 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG [...] 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG "pod": "am-767f6b67b5-vbltz" 06:02:47 DEBUG }, 06:02:47 DEBUG "value": [ 06:02:47 DEBUG 1713589367, 06:02:47 DEBUG "2660.133333333333" 06:02:47 DEBUG ] 06:02:47 DEBUG } 06:02:47 DEBUG ] 06:02:47 DEBUG } 06:02:47 DEBUG } 06:02:47 INFO [http_cmd]: http status code OK 06:02:47 DEBUG --- status code --- 06:02:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:47 DEBUG --- http response --- 06:02:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:02:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589367" 06:02:47 INFO [http_cmd]: http status code OK 06:02:47 DEBUG --- status code --- 06:02:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:47 DEBUG --- http response --- 06:02:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:02:47 DEBUG ----- output ----- 06:02:47 DEBUG { 06:02:47 DEBUG "status": "success", 06:02:47 DEBUG "data": { 06:02:47 DEBUG "resultType": "vector", 06:02:47 DEBUG "result": [ 06:02:47 DEBUG { 06:02:47 DEBUG "metric": { 06:02:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:02:47 DEBUG }, 06:02:47 DEBUG "value": [ 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG [...] 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG 06:02:47 DEBUG "pod": "ds-cts-2" 06:02:47 DEBUG }, 06:02:47 DEBUG "value": [ 06:02:47 DEBUG 1713589367, 06:02:47 DEBUG "0" 06:02:47 DEBUG ] 06:02:47 DEBUG } 06:02:47 DEBUG ] 06:02:47 DEBUG } 06:02:47 DEBUG } 06:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589368" 06:02:48 INFO [http_cmd]: http status code OK 06:02:48 DEBUG --- status code --- 06:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:48 DEBUG --- http response --- 06:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:48 DEBUG ----- output ----- 06:02:48 DEBUG { 06:02:48 DEBUG "status": "success", 06:02:48 DEBUG "data": { 06:02:48 DEBUG "resultType": "vector", 06:02:48 DEBUG "result": [ 06:02:48 DEBUG { 06:02:48 DEBUG "metric": { 06:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG [...] 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 1713589368, 06:02:48 DEBUG "0" 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG } 06:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589368" 06:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589368" 06:02:48 INFO [http_cmd]: http status code OK 06:02:48 DEBUG --- status code --- 06:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:48 DEBUG --- http response --- 06:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:02:48 DEBUG ----- output ----- 06:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589368" 06:02:48 DEBUG { 06:02:48 DEBUG "status": "success", 06:02:48 DEBUG "data": { 06:02:48 DEBUG "resultType": "vector", 06:02:48 DEBUG "result": [ 06:02:48 DEBUG { 06:02:48 DEBUG "metric": { 06:02:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG [...] 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG "pod": "ds-cts-2" 06:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589368" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 1713589368, 06:02:48 DEBUG "0" 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG } 06:02:48 INFO [http_cmd]: http status code OK 06:02:48 DEBUG --- status code --- 06:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:48 DEBUG --- http response --- 06:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:48 DEBUG ----- output ----- 06:02:48 DEBUG { 06:02:48 DEBUG "status": "success", 06:02:48 DEBUG "data": { 06:02:48 DEBUG "resultType": "vector", 06:02:48 DEBUG "result": [ 06:02:48 DEBUG { 06:02:48 DEBUG "metric": { 06:02:48 DEBUG "instance": "10.142.0.107:9100" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG [...] 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG "instance": "10.142.0.43:9100" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 1713589368, 06:02:48 DEBUG "80901214.23333333" 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG } 06:02:48 INFO [http_cmd]: http status code OK 06:02:48 DEBUG --- status code --- 06:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:48 DEBUG --- http response --- 06:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:48 DEBUG ----- output ----- 06:02:48 DEBUG { 06:02:48 DEBUG "status": "success", 06:02:48 DEBUG "data": { 06:02:48 DEBUG "resultType": "vector", 06:02:48 DEBUG "result": [ 06:02:48 DEBUG { 06:02:48 DEBUG "metric": { 06:02:48 DEBUG "pod": "ds-idrepo-0" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG [...] 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG "pod": "ds-idrepo-2" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 1713589368, 06:02:48 DEBUG "0" 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG } 06:02:48 INFO [http_cmd]: http status code OK 06:02:48 DEBUG --- status code --- 06:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:48 DEBUG --- http response --- 06:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:48 DEBUG ----- output ----- 06:02:48 DEBUG { 06:02:48 DEBUG "status": "success", 06:02:48 DEBUG "data": { 06:02:48 DEBUG "resultType": "vector", 06:02:48 DEBUG "result": [ 06:02:48 DEBUG { 06:02:48 DEBUG "metric": { 06:02:48 DEBUG "instance": "10.142.0.107:9100" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG [...] 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG 06:02:48 DEBUG "instance": "10.142.0.43:9100" 06:02:48 DEBUG }, 06:02:48 DEBUG "value": [ 06:02:48 DEBUG 1713589368, 06:02:48 DEBUG "0.08699999999999478" 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG ] 06:02:48 DEBUG } 06:02:48 DEBUG } 06:02:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589370" 06:02:50 INFO [http_cmd]: http status code OK 06:02:50 DEBUG --- status code --- 06:02:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:50 DEBUG --- http response --- 06:02:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:50 DEBUG ----- output ----- 06:02:50 DEBUG { 06:02:50 DEBUG "status": "success", 06:02:50 DEBUG "data": { 06:02:50 DEBUG "resultType": "vector", 06:02:50 DEBUG "result": [ 06:02:50 DEBUG { 06:02:50 DEBUG "metric": { 06:02:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:02:50 DEBUG }, 06:02:50 DEBUG "value": [ 06:02:50 DEBUG 06:02:50 DEBUG 06:02:50 DEBUG 06:02:50 DEBUG [...] 06:02:50 DEBUG 06:02:50 DEBUG 06:02:50 DEBUG 06:02:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:02:50 DEBUG }, 06:02:50 DEBUG "value": [ 06:02:50 DEBUG 1713589370, 06:02:50 DEBUG "2671" 06:02:50 DEBUG ] 06:02:50 DEBUG } 06:02:50 DEBUG ] 06:02:50 DEBUG } 06:02:50 DEBUG } 06:02:50 INFO 06:02:50 INFO [loop_until]: kubectl --namespace=xlou top pods 06:02:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:02:50 INFO 06:02:50 INFO [loop_until]: kubectl --namespace=xlou top node 06:02:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:02:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589370" 06:02:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589370" 06:02:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589370" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589371" 06:02:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589371" 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [loop_until]: OK (rc = 0) 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG --- status code --- 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG --- status code --- 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG --- status code --- 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG --- status code --- 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 INFO [loop_until]: OK (rc = 0) 06:02:51 DEBUG --- stdout --- 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- status code --- 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG --- status code --- 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG --- stdout --- 06:02:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 6191Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7983m 26% 7385Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17334m 58% 11365Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3372Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18641m 62% 13768Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5169Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17435m 58% 11604Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19144m 64% 3012Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19553m 65% 10864Mi 9% gke-xlou-cdm-ds-0f601751-j488 8305m 27% 6960Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17222m 57% 15706Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8116m 27% 7596Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19314m 64% 13214Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8179m 51% 11185Mi 19% 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:51 DEBUG --- stderr --- 06:02:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17403m 8652Mi am-767f6b67b5-qcqz2 17145m 8714Mi am-767f6b67b5-vbltz 18573m 8649Mi ds-cts-0 19160m 10632Mi ds-cts-1 17367m 10738Mi ds-cts-2 18901m 10610Mi ds-idrepo-0 7693m 3714Mi ds-idrepo-1 7558m 3704Mi ds-idrepo-2 8100m 3626Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 12m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7837m 7379Mi 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:51 INFO [http_cmd]: http status code OK 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG 06:02:51 DEBUG --- stderr --- 06:02:51 DEBUG 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:02:51 DEBUG --- status code --- 06:02:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG { 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG --- http response --- 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG { 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG { 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG { 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "data": { 06:02:51 DEBUG { 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG { 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG { 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG { 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG ----- output ----- 06:02:51 DEBUG { 06:02:51 DEBUG "data": { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "data": { 06:02:51 DEBUG "data": { 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "data": { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "data": { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "status": "success", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "data": { 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG { 06:02:51 DEBUG "data": { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "data": { 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "data": { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG { 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "result": [ 06:02:51 DEBUG { 06:02:51 DEBUG { 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG { 06:02:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:02:51 DEBUG { 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "resultType": "vector", 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "metric": { 06:02:51 DEBUG { 06:02:51 DEBUG "metric": { 06:02:51 DEBUG { 06:02:51 DEBUG "pod": "ds-cts-0" 06:02:51 DEBUG "result": [ 06:02:51 DEBUG { 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "pod": "ds-idrepo-0" 06:02:51 DEBUG "metric": { 06:02:51 DEBUG }, 06:02:51 DEBUG "metric": { 06:02:51 DEBUG { 06:02:51 DEBUG "result": [ 06:02:51 DEBUG { 06:02:51 DEBUG { 06:02:51 DEBUG "pod": "ds-cts-0" 06:02:51 DEBUG "pod": "ds-idrepo-0" 06:02:51 DEBUG "result": [ 06:02:51 DEBUG "metric": { 06:02:51 DEBUG }, 06:02:51 DEBUG { 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "instance": "10.142.0.107:9100" 06:02:51 DEBUG "result": [ 06:02:51 DEBUG }, 06:02:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "pod": "ds-cts-0" 06:02:51 DEBUG "metric": { 06:02:51 DEBUG { 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "metric": { 06:02:51 DEBUG }, 06:02:51 DEBUG }, 06:02:51 DEBUG { 06:02:51 DEBUG "instance": "10.142.0.107:9100" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:02:51 DEBUG }, 06:02:51 DEBUG { 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG 06:02:51 DEBUG }, 06:02:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:02:51 DEBUG "metric": { 06:02:51 DEBUG "instance": "10.142.0.107:9100" 06:02:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "metric": { 06:02:51 DEBUG }, 06:02:51 DEBUG 06:02:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:02:51 DEBUG }, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "metric": { 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:02:51 DEBUG }, 06:02:51 DEBUG }, 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG }, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG "pod": "ds-idrepo-0" 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG }, 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG }, 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "pod": "ds-idrepo-2" 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG [...] 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:02:51 DEBUG 06:02:51 DEBUG "pod": "ds-idrepo-2" 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "pod": "ds-cts-2" 06:02:51 DEBUG }, 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "pod": "ds-cts-2" 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "instance": "10.142.0.43:9100" 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:02:51 DEBUG }, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG 06:02:51 DEBUG }, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "instance": "10.142.0.43:9100" 06:02:51 DEBUG 06:02:51 DEBUG }, 06:02:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:02:51 DEBUG 06:02:51 DEBUG }, 06:02:51 DEBUG }, 06:02:51 DEBUG "pod": "ds-cts-2" 06:02:51 DEBUG }, 06:02:51 DEBUG "pod": "ds-cts-2" 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:02:51 DEBUG 06:02:51 DEBUG "instance": "10.142.0.43:9100" 06:02:51 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG }, 06:02:51 DEBUG 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG "0" 06:02:51 DEBUG }, 06:02:51 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 06:02:51 DEBUG }, 06:02:51 DEBUG }, 06:02:51 DEBUG 1713589370, 06:02:51 DEBUG "0" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG ] 06:02:51 DEBUG "value": [ 06:02:51 DEBUG }, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "0.0007" 06:02:51 DEBUG ] 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG }, 06:02:51 DEBUG "0" 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG "80894233.5" 06:02:51 DEBUG "2671.9666666666667" 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "7835271168" 06:02:51 DEBUG 1713589370, 06:02:51 DEBUG } 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "value": [ 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG 1713589370, 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG "3143533892608" 06:02:51 DEBUG "value": [ 06:02:51 DEBUG ] 06:02:51 DEBUG "480234.5" 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG "0" 06:02:51 DEBUG ] 06:02:51 DEBUG "30" 06:02:51 DEBUG ] 06:02:51 DEBUG "2671.4666666666667" 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG "0.23556666666665554" 06:02:51 DEBUG "0" 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG 1713589371, 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG "0.092" 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG "0" 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG "8.02721240854333" 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG ] 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:02:51 DEBUG } 06:03:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589427" 06:03:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589427" 06:03:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589427" 06:03:47 INFO [http_cmd]: http status code OK 06:03:47 DEBUG --- status code --- 06:03:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:47 DEBUG --- http response --- 06:03:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:47 DEBUG ----- output ----- 06:03:47 DEBUG { 06:03:47 DEBUG "status": "success", 06:03:47 DEBUG "data": { 06:03:47 DEBUG "resultType": "vector", 06:03:47 DEBUG "result": [ 06:03:47 DEBUG { 06:03:47 DEBUG "metric": { 06:03:47 DEBUG "pod": "am-767f6b67b5-9zw22" 06:03:47 DEBUG }, 06:03:47 DEBUG "value": [ 06:03:47 DEBUG 06:03:47 DEBUG 06:03:47 DEBUG 06:03:47 DEBUG [...] 06:03:47 DEBUG 06:03:47 DEBUG 06:03:47 DEBUG 06:03:47 DEBUG "pod": "am-767f6b67b5-vbltz" 06:03:47 DEBUG }, 06:03:47 DEBUG "value": [ 06:03:47 DEBUG 1713589427, 06:03:47 DEBUG "2672.6666666666665" 06:03:47 DEBUG ] 06:03:47 DEBUG } 06:03:47 DEBUG ] 06:03:47 DEBUG } 06:03:47 DEBUG } 06:03:47 ERROR [http_cmd]: ERROR 06:03:47 ERROR --- status code --- 06:03:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:03:47 ERROR --- http response --- 06:03:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:03:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:03:47 INFO [http_cmd]: http status code OK 06:03:47 DEBUG --- status code --- 06:03:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:47 DEBUG --- http response --- 06:03:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:03:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589427" 06:03:48 INFO [http_cmd]: http status code OK 06:03:48 DEBUG --- status code --- 06:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:48 DEBUG --- http response --- 06:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:03:48 DEBUG ----- output ----- 06:03:48 DEBUG { 06:03:48 DEBUG "status": "success", 06:03:48 DEBUG "data": { 06:03:48 DEBUG "resultType": "vector", 06:03:48 DEBUG "result": [ 06:03:48 DEBUG { 06:03:48 DEBUG "metric": { 06:03:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG [...] 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 1713589427, 06:03:48 DEBUG "632226002.6433265" 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG } 06:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589428" 06:03:48 INFO [http_cmd]: http status code OK 06:03:48 DEBUG --- status code --- 06:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:48 DEBUG --- http response --- 06:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:48 DEBUG ----- output ----- 06:03:48 DEBUG { 06:03:48 DEBUG "status": "success", 06:03:48 DEBUG "data": { 06:03:48 DEBUG "resultType": "vector", 06:03:48 DEBUG "result": [ 06:03:48 DEBUG { 06:03:48 DEBUG "metric": { 06:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG [...] 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 1713589428, 06:03:48 DEBUG "0" 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG } 06:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589428" 06:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589428" 06:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589428" 06:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589428" 06:03:48 INFO [http_cmd]: http status code OK 06:03:48 INFO [http_cmd]: http status code OK 06:03:48 DEBUG --- status code --- 06:03:48 DEBUG --- status code --- 06:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:48 DEBUG --- http response --- 06:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:48 DEBUG --- http response --- 06:03:48 DEBUG ----- output ----- 06:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:03:48 DEBUG { 06:03:48 DEBUG "status": "success", 06:03:48 DEBUG "data": { 06:03:48 DEBUG "resultType": "vector", 06:03:48 DEBUG "result": [ 06:03:48 DEBUG { 06:03:48 DEBUG "metric": { 06:03:48 DEBUG "instance": "10.142.0.107:9100" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG [...] 06:03:48 DEBUG 06:03:48 DEBUG ----- output ----- 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG "instance": "10.142.0.43:9100" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 1713589428, 06:03:48 DEBUG "80975925.13333333" 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG } 06:03:48 DEBUG { 06:03:48 DEBUG "status": "success", 06:03:48 DEBUG "data": { 06:03:48 DEBUG "resultType": "vector", 06:03:48 DEBUG "result": [ 06:03:48 DEBUG { 06:03:48 DEBUG "metric": { 06:03:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG [...] 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG "pod": "ds-cts-2" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 1713589428, 06:03:48 DEBUG "19.612998481961085" 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG } 06:03:48 INFO [http_cmd]: http status code OK 06:03:48 DEBUG --- status code --- 06:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:48 DEBUG --- http response --- 06:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:48 DEBUG ----- output ----- 06:03:48 DEBUG { 06:03:48 DEBUG "status": "success", 06:03:48 DEBUG "data": { 06:03:48 DEBUG "resultType": "vector", 06:03:48 DEBUG "result": [ 06:03:48 DEBUG { 06:03:48 DEBUG "metric": { 06:03:48 DEBUG "instance": "10.142.0.107:9100" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 INFO [http_cmd]: http status code OK 06:03:48 DEBUG 06:03:48 DEBUG [...] 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG "instance": "10.142.0.43:9100" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 1713589428, 06:03:48 DEBUG "0.08333333333332953" 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG } 06:03:48 DEBUG --- status code --- 06:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:48 DEBUG --- http response --- 06:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:48 DEBUG ----- output ----- 06:03:48 DEBUG { 06:03:48 DEBUG "status": "success", 06:03:48 DEBUG "data": { 06:03:48 DEBUG "resultType": "vector", 06:03:48 DEBUG "result": [ 06:03:48 DEBUG { 06:03:48 DEBUG "metric": { 06:03:48 DEBUG "pod": "ds-idrepo-0" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG [...] 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG 06:03:48 DEBUG "pod": "ds-idrepo-2" 06:03:48 DEBUG }, 06:03:48 DEBUG "value": [ 06:03:48 DEBUG 1713589428, 06:03:48 DEBUG "0" 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG ] 06:03:48 DEBUG } 06:03:48 DEBUG } 06:03:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589430" 06:03:50 INFO [http_cmd]: http status code OK 06:03:50 DEBUG --- status code --- 06:03:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:50 DEBUG --- http response --- 06:03:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:50 DEBUG ----- output ----- 06:03:50 DEBUG { 06:03:50 DEBUG "status": "success", 06:03:50 DEBUG "data": { 06:03:50 DEBUG "resultType": "vector", 06:03:50 DEBUG "result": [ 06:03:50 DEBUG { 06:03:50 DEBUG "metric": { 06:03:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:03:50 DEBUG }, 06:03:50 DEBUG "value": [ 06:03:50 DEBUG 06:03:50 DEBUG 06:03:50 DEBUG 06:03:50 DEBUG [...] 06:03:50 DEBUG 06:03:50 DEBUG 06:03:50 DEBUG 06:03:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:03:50 DEBUG }, 06:03:50 DEBUG "value": [ 06:03:50 DEBUG 1713589430, 06:03:50 DEBUG "2676.866666666667" 06:03:50 DEBUG ] 06:03:50 DEBUG } 06:03:50 DEBUG ] 06:03:50 DEBUG } 06:03:50 DEBUG } 06:03:51 INFO 06:03:51 INFO 06:03:51 INFO [loop_until]: kubectl --namespace=xlou top node 06:03:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:03:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:03:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589431" 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [loop_until]: OK (rc = 0) 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 DEBUG --- status code --- 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG --- stdout --- 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG --- status code --- 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 DEBUG --- status code --- 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG --- status code --- 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 DEBUG --- status code --- 06:03:51 INFO [loop_until]: OK (rc = 0) 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17696m 8652Mi am-767f6b67b5-qcqz2 17595m 8715Mi am-767f6b67b5-vbltz 18247m 8648Mi ds-cts-0 18704m 10645Mi ds-cts-1 17069m 10733Mi ds-cts-2 19291m 10598Mi ds-idrepo-0 7623m 3713Mi ds-idrepo-1 7876m 3705Mi ds-idrepo-2 7797m 3626Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7959m 7378Mi 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- stdout --- 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG --- stderr --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 INFO [http_cmd]: http status code OK 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6190Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7942m 26% 7382Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17359m 58% 11371Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19044m 63% 13776Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5171Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17504m 58% 11601Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19210m 64% 3006Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19599m 65% 10864Mi 9% gke-xlou-cdm-ds-0f601751-j488 8284m 27% 6948Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17091m 57% 15706Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8061m 27% 7599Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 19667m 65% 13207Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8181m 51% 11183Mi 19% 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:51 DEBUG 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG --- status code --- 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:51 DEBUG --- stderr --- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:03:51 DEBUG { 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG { 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG { 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG { 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG { 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:03:51 DEBUG --- http response --- 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG { 06:03:51 DEBUG "data": { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "data": { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG ----- output ----- 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "data": { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "data": { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "result": [ 06:03:51 DEBUG { 06:03:51 DEBUG "data": { 06:03:51 DEBUG "data": { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "result": [ 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "result": [ 06:03:51 DEBUG { 06:03:51 DEBUG "result": [ 06:03:51 DEBUG { 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "pod": "ds-idrepo-0" 06:03:51 DEBUG "data": { 06:03:51 DEBUG "status": "success", 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "result": [ 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG "result": [ 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG { 06:03:51 DEBUG "result": [ 06:03:51 DEBUG }, 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG "data": { 06:03:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:03:51 DEBUG "pod": "ds-cts-0" 06:03:51 DEBUG { 06:03:51 DEBUG { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "pod": "ds-idrepo-0" 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "pod": "ds-idrepo-0" 06:03:51 DEBUG "metric": { 06:03:51 DEBUG { 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "resultType": "vector", 06:03:51 DEBUG }, 06:03:51 DEBUG }, 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "metric": { 06:03:51 DEBUG "instance": "10.142.0.107:9100" 06:03:51 DEBUG }, 06:03:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:03:51 DEBUG "instance": "10.142.0.107:9100" 06:03:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:03:51 DEBUG }, 06:03:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:03:51 DEBUG }, 06:03:51 DEBUG "pod": "ds-cts-0" 06:03:51 DEBUG "metric": { 06:03:51 DEBUG 06:03:51 DEBUG { 06:03:51 DEBUG "result": [ 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG "instance": "10.142.0.107:9100" 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "metric": { 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG }, 06:03:51 DEBUG "pod": "ds-cts-0" 06:03:51 DEBUG 06:03:51 DEBUG "metric": { 06:03:51 DEBUG { 06:03:51 DEBUG 06:03:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:03:51 DEBUG "metric": { 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG [...] 06:03:51 DEBUG }, 06:03:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG "pod": "ds-idrepo-2" 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG "pod": "ds-cts-2" 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG [...] 06:03:51 DEBUG 06:03:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG "instance": "10.142.0.43:9100" 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:03:51 DEBUG 06:03:51 DEBUG "pod": "ds-cts-2" 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG "pod": "ds-idrepo-2" 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG [...] 06:03:51 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 06:03:51 DEBUG }, 06:03:51 DEBUG 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG "instance": "10.142.0.43:9100" 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "pod": "ds-cts-2" 06:03:51 DEBUG "pod": "ds-cts-2" 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "0" 06:03:51 DEBUG 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:03:51 DEBUG "0" 06:03:51 DEBUG }, 06:03:51 DEBUG "3147162530816" 06:03:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:03:51 DEBUG "instance": "10.142.0.43:9100" 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG }, 06:03:51 DEBUG }, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG ] 06:03:51 DEBUG 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG }, 06:03:51 DEBUG ] 06:03:51 DEBUG "value": [ 06:03:51 DEBUG ] 06:03:51 DEBUG }, 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG "0" 06:03:51 DEBUG "480933.5" 06:03:51 DEBUG "value": [ 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG } 06:03:51 DEBUG 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG "2676.766666666667" 06:03:51 DEBUG "value": [ 06:03:51 DEBUG } 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG } 06:03:51 DEBUG "value": [ 06:03:51 DEBUG "7834382336" 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG "2676.633333333333" 06:03:51 DEBUG ] 06:03:51 DEBUG ] 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG "0.0007333333333333333" 06:03:51 DEBUG ] 06:03:51 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 06:03:51 DEBUG "0" 06:03:51 DEBUG "0" 06:03:51 DEBUG ] 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG ] 06:03:51 DEBUG "0.2253666666667404" 06:03:51 DEBUG ] 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG ] 06:03:51 DEBUG "80969060.8" 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG "24.53333333333333" 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG }, 06:03:51 DEBUG ] 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG "0.092" 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG "8.036329557443434" 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG ] 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG "value": [ 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG 1713589431, 06:03:51 DEBUG ] 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG "0" 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG } 06:03:51 DEBUG ] 06:03:51 DEBUG } 06:03:51 DEBUG } 06:04:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589487" 06:04:47 INFO [http_cmd]: http status code OK 06:04:47 DEBUG --- status code --- 06:04:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:47 DEBUG --- http response --- 06:04:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:04:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589487" 06:04:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589487" 06:04:48 ERROR [http_cmd]: ERROR 06:04:48 INFO [http_cmd]: http status code OK 06:04:48 ERROR --- status code --- 06:04:48 DEBUG --- status code --- 06:04:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:48 ERROR --- http response --- 06:04:48 DEBUG --- http response --- 06:04:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:04:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:48 DEBUG ----- output ----- 06:04:48 DEBUG { 06:04:48 DEBUG "status": "success", 06:04:48 DEBUG "data": { 06:04:48 DEBUG "resultType": "vector", 06:04:48 DEBUG "result": [ 06:04:48 DEBUG { 06:04:48 DEBUG "metric": { 06:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG [...] 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 1713589487, 06:04:48 DEBUG "2674.633333333333" 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG } 06:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589488" 06:04:48 INFO [http_cmd]: http status code OK 06:04:48 DEBUG --- status code --- 06:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:48 DEBUG --- http response --- 06:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:04:48 DEBUG ----- output ----- 06:04:48 DEBUG { 06:04:48 DEBUG "status": "success", 06:04:48 DEBUG "data": { 06:04:48 DEBUG "resultType": "vector", 06:04:48 DEBUG "result": [ 06:04:48 DEBUG { 06:04:48 DEBUG "metric": { 06:04:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG [...] 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 1713589488, 06:04:48 DEBUG "4414.933763406177" 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG } 06:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589488" 06:04:48 INFO [http_cmd]: http status code OK 06:04:48 DEBUG --- status code --- 06:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:48 DEBUG --- http response --- 06:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:48 DEBUG ----- output ----- 06:04:48 DEBUG { 06:04:48 DEBUG "status": "success", 06:04:48 DEBUG "data": { 06:04:48 DEBUG "resultType": "vector", 06:04:48 DEBUG "result": [ 06:04:48 DEBUG { 06:04:48 DEBUG "metric": { 06:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG [...] 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 1713589488, 06:04:48 DEBUG "0" 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG } 06:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589488" 06:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589488" 06:04:48 INFO [http_cmd]: http status code OK 06:04:48 DEBUG --- status code --- 06:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:48 DEBUG --- http response --- 06:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:48 DEBUG ----- output ----- 06:04:48 DEBUG { 06:04:48 DEBUG "status": "success", 06:04:48 DEBUG "data": { 06:04:48 DEBUG "resultType": "vector", 06:04:48 DEBUG "result": [ 06:04:48 DEBUG { 06:04:48 DEBUG "metric": { 06:04:48 DEBUG "instance": "10.142.0.107:9100" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG [...] 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG "instance": "10.142.0.43:9100" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 1713589488, 06:04:48 DEBUG "80681038.93333334" 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG } 06:04:48 INFO [http_cmd]: http status code OK 06:04:48 DEBUG --- status code --- 06:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:48 DEBUG --- http response --- 06:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:04:48 DEBUG ----- output ----- 06:04:48 DEBUG { 06:04:48 DEBUG "status": "success", 06:04:48 DEBUG "data": { 06:04:48 DEBUG "resultType": "vector", 06:04:48 DEBUG "result": [ 06:04:48 DEBUG { 06:04:48 DEBUG "metric": { 06:04:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG [...] 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 1713589488, 06:04:48 DEBUG "12.812030409499434" 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG } 06:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589488" 06:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589488" 06:04:48 INFO [http_cmd]: http status code OK 06:04:48 DEBUG --- status code --- 06:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:48 DEBUG --- http response --- 06:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:48 DEBUG ----- output ----- 06:04:48 DEBUG { 06:04:48 DEBUG "status": "success", 06:04:48 DEBUG "data": { 06:04:48 DEBUG "resultType": "vector", 06:04:48 DEBUG "result": [ 06:04:48 DEBUG { 06:04:48 DEBUG "metric": { 06:04:48 DEBUG "pod": "ds-idrepo-0" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG [...] 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG "pod": "ds-idrepo-2" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 1713589488, 06:04:48 DEBUG "0" 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG } 06:04:48 INFO [http_cmd]: http status code OK 06:04:48 DEBUG --- status code --- 06:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:48 DEBUG --- http response --- 06:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:48 DEBUG ----- output ----- 06:04:48 DEBUG { 06:04:48 DEBUG "status": "success", 06:04:48 DEBUG "data": { 06:04:48 DEBUG "resultType": "vector", 06:04:48 DEBUG "result": [ 06:04:48 DEBUG { 06:04:48 DEBUG "metric": { 06:04:48 DEBUG "instance": "10.142.0.107:9100" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG [...] 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG 06:04:48 DEBUG "instance": "10.142.0.43:9100" 06:04:48 DEBUG }, 06:04:48 DEBUG "value": [ 06:04:48 DEBUG 1713589488, 06:04:48 DEBUG "0.06666666666666288" 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG ] 06:04:48 DEBUG } 06:04:48 DEBUG } 06:04:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589490" 06:04:50 INFO [http_cmd]: http status code OK 06:04:50 DEBUG --- status code --- 06:04:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:50 DEBUG --- http response --- 06:04:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:50 DEBUG ----- output ----- 06:04:50 DEBUG { 06:04:50 DEBUG "status": "success", 06:04:50 DEBUG "data": { 06:04:50 DEBUG "resultType": "vector", 06:04:50 DEBUG "result": [ 06:04:50 DEBUG { 06:04:50 DEBUG "metric": { 06:04:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:04:50 DEBUG }, 06:04:50 DEBUG "value": [ 06:04:50 DEBUG 06:04:50 DEBUG 06:04:50 DEBUG 06:04:50 DEBUG [...] 06:04:50 DEBUG 06:04:50 DEBUG 06:04:50 DEBUG 06:04:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:50 DEBUG }, 06:04:50 DEBUG "value": [ 06:04:50 DEBUG 1713589490, 06:04:50 DEBUG "2653.0333333333333" 06:04:50 DEBUG ] 06:04:50 DEBUG } 06:04:50 DEBUG ] 06:04:50 DEBUG } 06:04:50 DEBUG } 06:04:51 INFO 06:04:51 INFO [loop_until]: kubectl --namespace=xlou top node 06:04:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:04:51 INFO 06:04:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:04:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:04:51 INFO [loop_until]: OK (rc = 0) 06:04:51 DEBUG --- stdout --- 06:04:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6198Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7900m 26% 7391Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17302m 57% 11370Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3376Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18405m 61% 13771Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5169Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17420m 58% 11593Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19074m 63% 3007Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 18699m 62% 10866Mi 9% gke-xlou-cdm-ds-0f601751-j488 8199m 27% 6950Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17021m 57% 15719Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 8007m 26% 7592Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 18857m 63% 13223Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8176m 51% 11183Mi 19% 06:04:51 DEBUG --- stderr --- 06:04:51 DEBUG 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589491" 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589491" 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [loop_until]: OK (rc = 0) 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 DEBUG --- status code --- 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 DEBUG --- status code --- 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- stdout --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- status code --- 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17565m 8651Mi am-767f6b67b5-qcqz2 16896m 8712Mi am-767f6b67b5-vbltz 18874m 8649Mi ds-cts-0 18600m 10630Mi ds-cts-1 16995m 10724Mi ds-cts-2 18956m 10602Mi ds-idrepo-0 7705m 3714Mi ds-idrepo-1 7657m 3705Mi ds-idrepo-2 7944m 3627Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 6m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8038m 7378Mi 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- stderr --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:51 DEBUG 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:04:51 INFO [http_cmd]: http status code OK 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG { 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG --- status code --- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG { 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG { 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "data": { 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG --- http response --- 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG { 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "data": { 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "data": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "data": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:04:51 DEBUG "data": { 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG "data": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "data": { 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG ----- output ----- 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "data": { 06:04:51 DEBUG { 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG { 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG { 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG { 06:04:51 DEBUG "result": [ 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG "status": "success", 06:04:51 DEBUG { 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "data": { 06:04:51 DEBUG { 06:04:51 DEBUG { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:04:51 DEBUG "pod": "ds-cts-0" 06:04:51 DEBUG { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "data": { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG { 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "instance": "10.142.0.107:9100" 06:04:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:04:51 DEBUG "pod": "ds-idrepo-0" 06:04:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:04:51 DEBUG "pod": "ds-idrepo-0" 06:04:51 DEBUG "resultType": "vector", 06:04:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:04:51 DEBUG "metric": { 06:04:51 DEBUG "instance": "10.142.0.107:9100" 06:04:51 DEBUG "pod": "ds-idrepo-0" 06:04:51 DEBUG "pod": "ds-cts-0" 06:04:51 DEBUG }, 06:04:51 DEBUG "instance": "10.142.0.107:9100" 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "pod": "ds-cts-0" 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG "result": [ 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "result": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG { 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG { 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "metric": { 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "metric": { 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG }, 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG [...] 06:04:51 DEBUG [...] 06:04:51 DEBUG }, 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG [...] 06:04:51 DEBUG "value": [ 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:04:51 DEBUG "pod": "ds-cts-2" 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "pod": "ds-idrepo-2" 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG 06:04:51 DEBUG "pod": "ds-cts-2" 06:04:51 DEBUG 06:04:51 DEBUG "instance": "10.142.0.43:9100" 06:04:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:51 DEBUG "pod": "ds-cts-2" 06:04:51 DEBUG [...] 06:04:51 DEBUG 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG 06:04:51 DEBUG "pod": "ds-cts-2" 06:04:51 DEBUG 06:04:51 DEBUG }, 06:04:51 DEBUG [...] 06:04:51 DEBUG "pod": "ds-idrepo-2" 06:04:51 DEBUG "instance": "10.142.0.43:9100" 06:04:51 DEBUG }, 06:04:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:51 DEBUG "instance": "10.142.0.43:9100" 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG 06:04:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:04:51 DEBUG }, 06:04:51 DEBUG "pod": "ds-cts-0" 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 06:04:51 DEBUG }, 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 06:04:51 DEBUG }, 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG }, 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "29.966666666666665" 06:04:51 DEBUG }, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG "0" 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG "value": [ 06:04:51 DEBUG "0" 06:04:51 DEBUG 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG ] 06:04:51 DEBUG "value": [ 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG "80676999.36666667" 06:04:51 DEBUG "2652.6" 06:04:51 DEBUG "0" 06:04:51 DEBUG "pod": "ds-cts-0" 06:04:51 DEBUG "480899.5" 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG "0.00013333333333333334" 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG "0" 06:04:51 DEBUG "0.1693666666667648" 06:04:51 DEBUG } 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG "3150738263040" 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG }, 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG "7834034176" 06:04:51 DEBUG "2652.9666666666667" 06:04:51 DEBUG ] 06:04:51 DEBUG "30312765.5530375" 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:04:51 DEBUG ] 06:04:51 DEBUG "0.092" 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG "value": [ 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG }, 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG "value": [ 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG "56463205.78989691" 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG 1713589491, 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG "8.034880524163839" 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG ] 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:51 DEBUG } 06:04:57 WARNING File /tmp/lodemon_data-20240420_053236/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [20/Apr/2024 06:04:57] "GET /monitoring/average?start_time=20240420_043547&stop_time=20240420_050257 HTTP/1.1" 200 - 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589548" 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 DEBUG --- status code --- 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589548" 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589548" 06:05:48 ERROR [http_cmd]: ERROR 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 ERROR --- status code --- 06:05:48 DEBUG --- status code --- 06:05:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 ERROR --- http response --- 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:05:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:48 DEBUG ----- output ----- 06:05:48 DEBUG { 06:05:48 DEBUG "status": "success", 06:05:48 DEBUG "data": { 06:05:48 DEBUG "resultType": "vector", 06:05:48 DEBUG "result": [ 06:05:48 DEBUG { 06:05:48 DEBUG "metric": { 06:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG [...] 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 1713589548, 06:05:48 DEBUG "2670.5" 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG } 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589548" 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 DEBUG --- status code --- 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:05:48 DEBUG ----- output ----- 06:05:48 DEBUG { 06:05:48 DEBUG "status": "success", 06:05:48 DEBUG "data": { 06:05:48 DEBUG "resultType": "vector", 06:05:48 DEBUG "result": [ 06:05:48 DEBUG { 06:05:48 DEBUG "metric": { 06:05:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG [...] 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG "pod": "ds-cts-2" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 1713589548, 06:05:48 DEBUG "357410939.45935494" 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG } 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589548" 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 DEBUG --- status code --- 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:48 DEBUG ----- output ----- 06:05:48 DEBUG { 06:05:48 DEBUG "status": "success", 06:05:48 DEBUG "data": { 06:05:48 DEBUG "resultType": "vector", 06:05:48 DEBUG "result": [ 06:05:48 DEBUG { 06:05:48 DEBUG "metric": { 06:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG [...] 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 1713589548, 06:05:48 DEBUG "0" 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG } 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589548" 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589548" 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 DEBUG --- status code --- 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:05:48 DEBUG ----- output ----- 06:05:48 DEBUG { 06:05:48 DEBUG "status": "success", 06:05:48 DEBUG "data": { 06:05:48 DEBUG "resultType": "vector", 06:05:48 DEBUG "result": [ 06:05:48 DEBUG { 06:05:48 DEBUG "metric": { 06:05:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG [...] 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG "pod": "ds-cts-2" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 1713589548, 06:05:48 DEBUG "8.201372394240789" 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG } 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 DEBUG --- status code --- 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:48 DEBUG ----- output ----- 06:05:48 DEBUG { 06:05:48 DEBUG "status": "success", 06:05:48 DEBUG "data": { 06:05:48 DEBUG "resultType": "vector", 06:05:48 DEBUG "result": [ 06:05:48 DEBUG { 06:05:48 DEBUG "metric": { 06:05:48 DEBUG "instance": "10.142.0.107:9100" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG [...] 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG "instance": "10.142.0.43:9100" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 1713589548, 06:05:48 DEBUG "33932180.46666667" 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG } 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589548" 06:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589548" 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 DEBUG --- status code --- 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:48 DEBUG ----- output ----- 06:05:48 DEBUG { 06:05:48 DEBUG "status": "success", 06:05:48 DEBUG "data": { 06:05:48 DEBUG "resultType": "vector", 06:05:48 DEBUG "result": [ 06:05:48 DEBUG { 06:05:48 DEBUG "metric": { 06:05:48 DEBUG "pod": "ds-idrepo-0" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG [...] 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG "pod": "ds-idrepo-2" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 1713589548, 06:05:48 DEBUG "0" 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG } 06:05:48 INFO [http_cmd]: http status code OK 06:05:48 DEBUG --- status code --- 06:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:48 DEBUG --- http response --- 06:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:48 DEBUG ----- output ----- 06:05:48 DEBUG { 06:05:48 DEBUG "status": "success", 06:05:48 DEBUG "data": { 06:05:48 DEBUG "resultType": "vector", 06:05:48 DEBUG "result": [ 06:05:48 DEBUG { 06:05:48 DEBUG "metric": { 06:05:48 DEBUG "instance": "10.142.0.107:9100" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG [...] 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG 06:05:48 DEBUG "instance": "10.142.0.43:9100" 06:05:48 DEBUG }, 06:05:48 DEBUG "value": [ 06:05:48 DEBUG 1713589548, 06:05:48 DEBUG "0.14833333333332913" 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG ] 06:05:48 DEBUG } 06:05:48 DEBUG } 06:05:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589550" 06:05:50 INFO [http_cmd]: http status code OK 06:05:50 DEBUG --- status code --- 06:05:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:50 DEBUG --- http response --- 06:05:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:50 DEBUG ----- output ----- 06:05:50 DEBUG { 06:05:50 DEBUG "status": "success", 06:05:50 DEBUG "data": { 06:05:50 DEBUG "resultType": "vector", 06:05:50 DEBUG "result": [ 06:05:50 DEBUG { 06:05:50 DEBUG "metric": { 06:05:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:05:50 DEBUG }, 06:05:50 DEBUG "value": [ 06:05:50 DEBUG 06:05:50 DEBUG 06:05:50 DEBUG 06:05:50 DEBUG [...] 06:05:50 DEBUG 06:05:50 DEBUG 06:05:50 DEBUG 06:05:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:50 DEBUG }, 06:05:50 DEBUG "value": [ 06:05:50 DEBUG 1713589550, 06:05:50 DEBUG "20.166666666666668" 06:05:50 DEBUG ] 06:05:50 DEBUG } 06:05:50 DEBUG ] 06:05:50 DEBUG } 06:05:50 DEBUG } 06:05:51 INFO 06:05:51 INFO [loop_until]: kubectl --namespace=xlou top node 06:05:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:05:51 INFO 06:05:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:05:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG { 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "data": { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "result": [ 06:05:51 DEBUG { 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "pod": "ds-cts-0" 06:05:51 DEBUG }, 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "pod": "ds-cts-2" 06:05:51 DEBUG }, 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "0" 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 INFO [loop_until]: OK (rc = 0) 06:05:51 DEBUG --- stdout --- 06:05:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6204Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 827m 2% 7385Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2721m 9% 11384Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8364m 28% 13780Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5177Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2797m 9% 11617Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 500m 1% 2973Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 3264m 10% 10884Mi 9% gke-xlou-cdm-ds-0f601751-j488 923m 3% 6937Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 7072m 23% 15715Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 865m 2% 7594Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8261m 27% 13218Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 2041m 12% 10795Mi 18% 06:05:51 DEBUG --- stderr --- 06:05:51 DEBUG 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589551" 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [loop_until]: OK (rc = 0) 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- stdout --- 06:05:51 DEBUG --- status code --- 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 DEBUG --- status code --- 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3082m 8650Mi am-767f6b67b5-qcqz2 3017m 8710Mi am-767f6b67b5-vbltz 3190m 8648Mi ds-cts-0 8243m 10604Mi ds-cts-1 7112m 10708Mi ds-cts-2 8169m 10578Mi ds-idrepo-0 748m 3713Mi ds-idrepo-1 807m 3705Mi ds-idrepo-2 847m 3619Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1176Mi idm-b85f9d978-wnhxb 3m 1235Mi lodemon-799cdc5bdb-7t5x7 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 2127m 6857Mi 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 INFO [http_cmd]: http status code OK 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- stderr --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG { 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- status code --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "data": { 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG { 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG { 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:51 DEBUG { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG { 06:05:51 DEBUG { 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG { 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG "data": { 06:05:51 DEBUG { 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "result": [ 06:05:51 DEBUG --- http response --- 06:05:51 DEBUG { 06:05:51 DEBUG { 06:05:51 DEBUG { 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG { 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG { 06:05:51 DEBUG { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "data": { 06:05:51 DEBUG { 06:05:51 DEBUG "data": { 06:05:51 DEBUG { 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "data": { 06:05:51 DEBUG "data": { 06:05:51 DEBUG "data": { 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "data": { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "data": { 06:05:51 DEBUG ----- output ----- 06:05:51 DEBUG "data": { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "data": { 06:05:51 DEBUG "data": { 06:05:51 DEBUG "data": { 06:05:51 DEBUG { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG { 06:05:51 DEBUG "data": { 06:05:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:05:51 DEBUG }, 06:05:51 DEBUG { 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "data": { 06:05:51 DEBUG "result": [ 06:05:51 DEBUG { 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "status": "success", 06:05:51 DEBUG { 06:05:51 DEBUG { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "instance": "10.142.0.107:9100" 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:05:51 DEBUG { 06:05:51 DEBUG { 06:05:51 DEBUG "pod": "ds-idrepo-0" 06:05:51 DEBUG "result": [ 06:05:51 DEBUG 06:05:51 DEBUG "result": [ 06:05:51 DEBUG "data": { 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "result": [ 06:05:51 DEBUG }, 06:05:51 DEBUG { 06:05:51 DEBUG "result": [ 06:05:51 DEBUG { 06:05:51 DEBUG { 06:05:51 DEBUG }, 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "metric": { 06:05:51 DEBUG }, 06:05:51 DEBUG { 06:05:51 DEBUG 06:05:51 DEBUG { 06:05:51 DEBUG "resultType": "vector", 06:05:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:05:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:05:51 DEBUG { 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "metric": { 06:05:51 DEBUG { 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "pod": "ds-cts-0" 06:05:51 DEBUG "instance": "10.142.0.107:9100" 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "metric": { 06:05:51 DEBUG 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "result": [ 06:05:51 DEBUG }, 06:05:51 DEBUG }, 06:05:51 DEBUG "metric": { 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "metric": { 06:05:51 DEBUG "instance": "10.142.0.107:9100" 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:05:51 DEBUG [...] 06:05:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:05:51 DEBUG { 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:05:51 DEBUG "pod": "ds-idrepo-0" 06:05:51 DEBUG 06:05:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "pod": "ds-cts-0" 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG "metric": { 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG }, 06:05:51 DEBUG [...] 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "pod": "ds-idrepo-0" 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "instance": "10.142.0.43:9100" 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG "value": [ 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG [...] 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG "pod": "ds-idrepo-2" 06:05:51 DEBUG 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG "instance": "10.142.0.43:9100" 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG "407969" 06:05:51 DEBUG 06:05:51 DEBUG [...] 06:05:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "7288111104" 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG }, 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "pod": "ds-cts-2" 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG "value": [ 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG 06:05:51 DEBUG ] 06:05:51 DEBUG "instance": "10.142.0.43:9100" 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:51 DEBUG 06:05:51 DEBUG 06:05:51 DEBUG } 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "value": [ 06:05:51 DEBUG }, 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:05:51 DEBUG "0.26116666666666316" 06:05:51 DEBUG "33923432.56666666" 06:05:51 DEBUG "pod": "ds-cts-2" 06:05:51 DEBUG 06:05:51 DEBUG } 06:05:51 DEBUG }, 06:05:51 DEBUG "0" 06:05:51 DEBUG }, 06:05:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:51 DEBUG 06:05:51 DEBUG } 06:05:51 DEBUG "0" 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG 06:05:51 DEBUG }, 06:05:51 DEBUG ] 06:05:51 DEBUG ] 06:05:51 DEBUG }, 06:05:51 DEBUG "pod": "ds-cts-2" 06:05:51 DEBUG }, 06:05:51 DEBUG "value": [ 06:05:51 DEBUG ] 06:05:51 DEBUG "value": [ 06:05:51 DEBUG }, 06:05:51 DEBUG 06:05:51 DEBUG ] 06:05:51 DEBUG "0" 06:05:51 DEBUG "0" 06:05:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:05:51 DEBUG "value": [ 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "value": [ 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "pod": "ds-idrepo-2" 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG ] 06:05:51 DEBUG }, 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG ] 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "3153045689344" 06:05:51 DEBUG "0.092" 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG }, 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG "value": [ 06:05:51 DEBUG "2.1589330267111193" 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "30" 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG "value": [ 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG "0" 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG "20.233333333333334" 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG "20.03333333333333" 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG 1713589551, 06:05:51 DEBUG "0" 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG ] 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG ] 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:05:51 DEBUG } 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589608" 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589608" 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:48 DEBUG ----- output ----- 06:06:48 DEBUG { 06:06:48 DEBUG "status": "success", 06:06:48 DEBUG "data": { 06:06:48 DEBUG "resultType": "vector", 06:06:48 DEBUG "result": [ 06:06:48 DEBUG { 06:06:48 DEBUG "metric": { 06:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG [...] 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 1713589608, 06:06:48 DEBUG "2676.7666666666664" 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG } 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589608" 06:06:48 ERROR [http_cmd]: ERROR 06:06:48 ERROR --- status code --- 06:06:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:06:48 ERROR --- http response --- 06:06:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:06:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589608" 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:06:48 DEBUG ----- output ----- 06:06:48 DEBUG { 06:06:48 DEBUG "status": "success", 06:06:48 DEBUG "data": { 06:06:48 DEBUG "resultType": "vector", 06:06:48 DEBUG "result": [ 06:06:48 DEBUG { 06:06:48 DEBUG "metric": { 06:06:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG [...] 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 1713589608, 06:06:48 DEBUG "4941110.331671896" 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG } 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589608" 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589608" 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589608" 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:06:48 DEBUG ----- output ----- 06:06:48 DEBUG { 06:06:48 DEBUG "status": "success", 06:06:48 DEBUG "data": { 06:06:48 DEBUG "resultType": "vector", 06:06:48 DEBUG "result": [ 06:06:48 DEBUG { 06:06:48 DEBUG "metric": { 06:06:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG [...] 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 1713589608, 06:06:48 DEBUG "8.002679563596613" 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG } 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:48 DEBUG ----- output ----- 06:06:48 DEBUG { 06:06:48 DEBUG "status": "success", 06:06:48 DEBUG "data": { 06:06:48 DEBUG "resultType": "vector", 06:06:48 DEBUG "result": [ 06:06:48 DEBUG { 06:06:48 DEBUG "metric": { 06:06:48 DEBUG "instance": "10.142.0.107:9100" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG [...] 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG "instance": "10.142.0.43:9100" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 1713589608, 06:06:48 DEBUG "49533348.03333333" 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG } 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:48 DEBUG ----- output ----- 06:06:48 DEBUG { 06:06:48 DEBUG "status": "success", 06:06:48 DEBUG "data": { 06:06:48 DEBUG "resultType": "vector", 06:06:48 DEBUG "result": [ 06:06:48 DEBUG { 06:06:48 DEBUG "metric": { 06:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG [...] 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 1713589608, 06:06:48 DEBUG "0" 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG } 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589608" 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:48 DEBUG ----- output ----- 06:06:48 DEBUG { 06:06:48 DEBUG "status": "success", 06:06:48 DEBUG "data": { 06:06:48 DEBUG "resultType": "vector", 06:06:48 DEBUG "result": [ 06:06:48 DEBUG { 06:06:48 DEBUG "metric": { 06:06:48 DEBUG "instance": "10.142.0.107:9100" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG [...] 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG "instance": "10.142.0.43:9100" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 1713589608, 06:06:48 DEBUG "0.07166666666667537" 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG } 06:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589608" 06:06:48 INFO [http_cmd]: http status code OK 06:06:48 DEBUG --- status code --- 06:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:48 DEBUG --- http response --- 06:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:48 DEBUG ----- output ----- 06:06:48 DEBUG { 06:06:48 DEBUG "status": "success", 06:06:48 DEBUG "data": { 06:06:48 DEBUG "resultType": "vector", 06:06:48 DEBUG "result": [ 06:06:48 DEBUG { 06:06:48 DEBUG "metric": { 06:06:48 DEBUG "pod": "ds-idrepo-0" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG [...] 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG 06:06:48 DEBUG "pod": "ds-idrepo-2" 06:06:48 DEBUG }, 06:06:48 DEBUG "value": [ 06:06:48 DEBUG 1713589608, 06:06:48 DEBUG "0" 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG ] 06:06:48 DEBUG } 06:06:48 DEBUG } 06:06:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589610" 06:06:50 INFO [http_cmd]: http status code OK 06:06:50 DEBUG --- status code --- 06:06:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:50 DEBUG --- http response --- 06:06:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:50 DEBUG ----- output ----- 06:06:50 DEBUG { 06:06:50 DEBUG "status": "success", 06:06:50 DEBUG "data": { 06:06:50 DEBUG "resultType": "vector", 06:06:50 DEBUG "result": [ 06:06:50 DEBUG { 06:06:50 DEBUG "metric": { 06:06:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:06:50 DEBUG }, 06:06:50 DEBUG "value": [ 06:06:50 DEBUG 06:06:50 DEBUG 06:06:50 DEBUG 06:06:50 DEBUG [...] 06:06:50 DEBUG 06:06:50 DEBUG 06:06:50 DEBUG 06:06:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:06:50 DEBUG }, 06:06:50 DEBUG "value": [ 06:06:50 DEBUG 1713589610, 06:06:50 DEBUG "1749.5333333333333" 06:06:50 DEBUG ] 06:06:50 DEBUG } 06:06:50 DEBUG ] 06:06:50 DEBUG } 06:06:50 DEBUG } 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- http response --- 06:06:51 INFO 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:06:51 INFO [loop_until]: kubectl --namespace=xlou top node 06:06:51 DEBUG ----- output ----- 06:06:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:06:51 DEBUG { 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "data": { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "result": [ 06:06:51 DEBUG { 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "pod": "ds-cts-0" 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "pod": "ds-cts-2" 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "0.0000666666666666667" 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 INFO 06:06:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:06:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:06:51 INFO [loop_until]: OK (rc = 0) 06:06:51 DEBUG --- stdout --- 06:06:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6207Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5798m 19% 7397Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11667m 39% 11373Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3364Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 13900m 46% 13769Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5187Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11831m 39% 11596Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 10330m 34% 3694Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 13890m 46% 10855Mi 9% gke-xlou-cdm-ds-0f601751-j488 5504m 18% 6942Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 12515m 41% 15719Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5748m 19% 7600Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 14922m 49% 13207Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8142m 51% 11153Mi 19% 06:06:51 DEBUG --- stderr --- 06:06:51 DEBUG 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG --- status code --- 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589611" 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 DEBUG --- http response --- 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589611" 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 DEBUG --- status code --- 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 DEBUG --- status code --- 06:06:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589611" 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 INFO [loop_until]: OK (rc = 0) 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG --- status code --- 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- status code --- 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG --- stdout --- 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG --- status code --- 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG ----- output ----- 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- status code --- 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 13452m 8655Mi am-767f6b67b5-qcqz2 13563m 8710Mi am-767f6b67b5-vbltz 12929m 8649Mi ds-cts-0 14951m 10628Mi ds-cts-1 13526m 10729Mi ds-cts-2 15863m 10588Mi ds-idrepo-0 5562m 3714Mi ds-idrepo-1 5044m 3705Mi ds-idrepo-2 6028m 3618Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7853m 7494Mi 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG { 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 INFO [http_cmd]: http status code OK 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG --- stderr --- 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG --- status code --- 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:06:51 DEBUG 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG "data": { 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG { 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG { 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "result": [ 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG --- http response --- 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG { 06:06:51 DEBUG "data": { 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "data": { 06:06:51 DEBUG { 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "data": { 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:51 DEBUG { 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "data": { 06:06:51 DEBUG "data": { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG { 06:06:51 DEBUG "data": { 06:06:51 DEBUG "data": { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:06:51 DEBUG "data": { 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "data": { 06:06:51 DEBUG ----- output ----- 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "data": { 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "data": { 06:06:51 DEBUG }, 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG { 06:06:51 DEBUG "data": { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "result": [ 06:06:51 DEBUG { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "data": { 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "result": [ 06:06:51 DEBUG { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "status": "success", 06:06:51 DEBUG "metric": { 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG { 06:06:51 DEBUG { 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "result": [ 06:06:51 DEBUG 06:06:51 DEBUG { 06:06:51 DEBUG "data": { 06:06:51 DEBUG "result": [ 06:06:51 DEBUG { 06:06:51 DEBUG "data": { 06:06:51 DEBUG "instance": "10.142.0.107:9100" 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:06:51 DEBUG { 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:06:51 DEBUG { 06:06:51 DEBUG 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG { 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "resultType": "vector", 06:06:51 DEBUG }, 06:06:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:06:51 DEBUG "instance": "10.142.0.107:9100" 06:06:51 DEBUG }, 06:06:51 DEBUG "metric": { 06:06:51 DEBUG { 06:06:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:06:51 DEBUG "pod": "ds-idrepo-0" 06:06:51 DEBUG }, 06:06:51 DEBUG "metric": { 06:06:51 DEBUG 06:06:51 DEBUG "instance": "10.142.0.107:9100" 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "metric": { 06:06:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:06:51 DEBUG "result": [ 06:06:51 DEBUG "value": [ 06:06:51 DEBUG }, 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "pod": "ds-cts-0" 06:06:51 DEBUG "metric": { 06:06:51 DEBUG }, 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "pod": "ds-idrepo-0" 06:06:51 DEBUG [...] 06:06:51 DEBUG }, 06:06:51 DEBUG { 06:06:51 DEBUG "pod": "ds-cts-0" 06:06:51 DEBUG }, 06:06:51 DEBUG { 06:06:51 DEBUG 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "metric": { 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "metric": { 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "value": [ 06:06:51 DEBUG }, 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "pod": "ds-idrepo-0" 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:06:51 DEBUG 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "instance": "10.142.0.43:9100" 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "pod": "ds-cts-2" 06:06:51 DEBUG 06:06:51 DEBUG "324847.5" 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:06:51 DEBUG }, 06:06:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:06:51 DEBUG "instance": "10.142.0.43:9100" 06:06:51 DEBUG }, 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:06:51 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:06:51 DEBUG }, 06:06:51 DEBUG 06:06:51 DEBUG ] 06:06:51 DEBUG "instance": "10.142.0.43:9100" 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG }, 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "pod": "ds-cts-2" 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "pod": "ds-idrepo-2" 06:06:51 DEBUG } 06:06:51 DEBUG }, 06:06:51 DEBUG 06:06:51 DEBUG [...] 06:06:51 DEBUG "pod": "ds-idrepo-2" 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "49525329.46666667" 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG }, 06:06:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG }, 06:06:51 DEBUG ] 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "pod": "ds-cts-2" 06:06:51 DEBUG 06:06:51 DEBUG }, 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG ] 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "0.14779999999995538" 06:06:51 DEBUG "value": [ 06:06:51 DEBUG }, 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "0" 06:06:51 DEBUG "value": [ 06:06:51 DEBUG } 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "3155725702144" 06:06:51 DEBUG }, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "8.03418576022562" 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG } 06:06:51 DEBUG "7957299200" 06:06:51 DEBUG ] 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "value": [ 06:06:51 DEBUG "126.02301396837116" 06:06:51 DEBUG "0" 06:06:51 DEBUG ] 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG } 06:06:51 DEBUG 06:06:51 DEBUG ] 06:06:51 DEBUG "value": [ 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG ] 06:06:51 DEBUG "0.092" 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG "30" 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG "0" 06:06:51 DEBUG 06:06:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG "0" 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG "1749.8666666666666" 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG }, 06:06:51 DEBUG "0" 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG "value": [ 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG 1713589611, 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG "1747.0333333333333" 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:06:51 DEBUG ] 06:06:51 DEBUG } 06:06:51 DEBUG } 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589668" 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 DEBUG --- status code --- 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589668" 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 DEBUG --- status code --- 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:48 DEBUG ----- output ----- 06:07:48 DEBUG { 06:07:48 DEBUG "status": "success", 06:07:48 DEBUG "data": { 06:07:48 DEBUG "resultType": "vector", 06:07:48 DEBUG "result": [ 06:07:48 DEBUG { 06:07:48 DEBUG "metric": { 06:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG [...] 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 1713589668, 06:07:48 DEBUG "2652.6666666666665" 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG } 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589668" 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589668" 06:07:48 ERROR [http_cmd]: ERROR 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 ERROR --- status code --- 06:07:48 DEBUG --- status code --- 06:07:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 ERROR --- http response --- 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:07:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:07:48 DEBUG ----- output ----- 06:07:48 DEBUG { 06:07:48 DEBUG "status": "success", 06:07:48 DEBUG "data": { 06:07:48 DEBUG "resultType": "vector", 06:07:48 DEBUG "result": [ 06:07:48 DEBUG { 06:07:48 DEBUG "metric": { 06:07:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG [...] 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG "pod": "ds-cts-0" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 1713589668, 06:07:48 DEBUG "307525.95953313087" 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG } 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589668" 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 DEBUG --- status code --- 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:48 DEBUG ----- output ----- 06:07:48 DEBUG { 06:07:48 DEBUG "status": "success", 06:07:48 DEBUG "data": { 06:07:48 DEBUG "resultType": "vector", 06:07:48 DEBUG "result": [ 06:07:48 DEBUG { 06:07:48 DEBUG "metric": { 06:07:48 DEBUG "instance": "10.142.0.107:9100" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG [...] 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG "instance": "10.142.0.43:9100" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 1713589668, 06:07:48 DEBUG "49039534.33333333" 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG } 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589668" 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589668" 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 DEBUG --- status code --- 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:07:48 DEBUG ----- output ----- 06:07:48 DEBUG { 06:07:48 DEBUG "status": "success", 06:07:48 DEBUG "data": { 06:07:48 DEBUG "resultType": "vector", 06:07:48 DEBUG "result": [ 06:07:48 DEBUG { 06:07:48 DEBUG "metric": { 06:07:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG [...] 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG "pod": "ds-cts-0" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 1713589668, 06:07:48 DEBUG "12.338908967837147" 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG } 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 DEBUG --- status code --- 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:48 DEBUG ----- output ----- 06:07:48 DEBUG { 06:07:48 DEBUG "status": "success", 06:07:48 DEBUG "data": { 06:07:48 DEBUG "resultType": "vector", 06:07:48 DEBUG "result": [ 06:07:48 DEBUG { 06:07:48 DEBUG "metric": { 06:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG [...] 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 1713589668, 06:07:48 DEBUG "0" 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG } 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589668" 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 DEBUG --- status code --- 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:48 DEBUG ----- output ----- 06:07:48 DEBUG { 06:07:48 DEBUG "status": "success", 06:07:48 DEBUG "data": { 06:07:48 DEBUG "resultType": "vector", 06:07:48 DEBUG "result": [ 06:07:48 DEBUG { 06:07:48 DEBUG "metric": { 06:07:48 DEBUG "instance": "10.142.0.107:9100" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG [...] 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG "instance": "10.142.0.43:9100" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 1713589668, 06:07:48 DEBUG "0.10933333333331535" 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG } 06:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589668" 06:07:48 INFO [http_cmd]: http status code OK 06:07:48 DEBUG --- status code --- 06:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:48 DEBUG --- http response --- 06:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:48 DEBUG ----- output ----- 06:07:48 DEBUG { 06:07:48 DEBUG "status": "success", 06:07:48 DEBUG "data": { 06:07:48 DEBUG "resultType": "vector", 06:07:48 DEBUG "result": [ 06:07:48 DEBUG { 06:07:48 DEBUG "metric": { 06:07:48 DEBUG "pod": "ds-idrepo-0" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG [...] 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG 06:07:48 DEBUG "pod": "ds-idrepo-2" 06:07:48 DEBUG }, 06:07:48 DEBUG "value": [ 06:07:48 DEBUG 1713589668, 06:07:48 DEBUG "0" 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG ] 06:07:48 DEBUG } 06:07:48 DEBUG } 06:07:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589670" 06:07:50 INFO [http_cmd]: http status code OK 06:07:50 DEBUG --- status code --- 06:07:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:50 DEBUG --- http response --- 06:07:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:50 DEBUG ----- output ----- 06:07:50 DEBUG { 06:07:50 DEBUG "status": "success", 06:07:50 DEBUG "data": { 06:07:50 DEBUG "resultType": "vector", 06:07:50 DEBUG "result": [ 06:07:50 DEBUG { 06:07:50 DEBUG "metric": { 06:07:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:07:50 DEBUG }, 06:07:50 DEBUG "value": [ 06:07:50 DEBUG 06:07:50 DEBUG 06:07:50 DEBUG 06:07:50 DEBUG [...] 06:07:50 DEBUG 06:07:50 DEBUG 06:07:50 DEBUG 06:07:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:07:50 DEBUG }, 06:07:50 DEBUG "value": [ 06:07:50 DEBUG 1713589670, 06:07:50 DEBUG "1341.9333333333334" 06:07:50 DEBUG ] 06:07:50 DEBUG } 06:07:50 DEBUG ] 06:07:50 DEBUG } 06:07:50 DEBUG } 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG { 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG "data": { 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "result": [ 06:07:51 DEBUG { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "pod": "ds-cts-0" 06:07:51 DEBUG }, 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "pod": "ds-cts-2" 06:07:51 DEBUG }, 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG "0.00003333333333333336" 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 INFO 06:07:51 INFO [loop_until]: kubectl --namespace=xlou top node 06:07:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:07:51 INFO [loop_until]: OK (rc = 0) 06:07:51 DEBUG --- stdout --- 06:07:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6206Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4302m 14% 7387Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9669m 32% 11371Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3365Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 12281m 41% 13771Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 5178Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9846m 32% 11593Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8617m 28% 3716Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 11211m 37% 10869Mi 9% gke-xlou-cdm-ds-0f601751-j488 4427m 14% 6934Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 11594m 38% 15690Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 4450m 14% 7599Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 12984m 43% 13177Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8179m 51% 11007Mi 18% 06:07:51 DEBUG --- stderr --- 06:07:51 DEBUG 06:07:51 INFO 06:07:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:07:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG --- status code --- 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:51 DEBUG --- http response --- 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG { 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG { 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG { 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG "data": { 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:51 DEBUG "data": { 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "data": { 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG { 06:07:51 DEBUG "result": [ 06:07:51 DEBUG { 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG { 06:07:51 DEBUG "data": { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:07:51 DEBUG }, 06:07:51 DEBUG "pod": "ds-cts-0" 06:07:51 DEBUG "result": [ 06:07:51 DEBUG }, 06:07:51 DEBUG "value": [ 06:07:51 DEBUG }, 06:07:51 DEBUG { 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "metric": { 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "pod": "ds-idrepo-0" 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG }, 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG "value": [ 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589671" 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 DEBUG 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG }, 06:07:51 DEBUG "value": [ 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG "pod": "ds-cts-2" 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:07:51 DEBUG 06:07:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589671" 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG --- status code --- 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG }, 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG }, 06:07:51 DEBUG 06:07:51 DEBUG "252197" 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG --- status code --- 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG "value": [ 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "pod": "ds-idrepo-2" 06:07:51 DEBUG ] 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG }, 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 INFO [loop_until]: OK (rc = 0) 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG "0" 06:07:51 DEBUG "0.092" 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG } 06:07:51 DEBUG "0" 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG --- stdout --- 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG { 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:07:51 DEBUG ] 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:07:51 INFO [http_cmd]: http status code OK 06:07:51 DEBUG { 06:07:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 10019m 8654Mi am-767f6b67b5-qcqz2 9841m 8709Mi am-767f6b67b5-vbltz 10557m 8647Mi ds-cts-0 12387m 10632Mi ds-cts-1 11402m 10713Mi ds-cts-2 12850m 10602Mi ds-idrepo-0 4135m 3715Mi ds-idrepo-1 4088m 3705Mi ds-idrepo-2 4203m 3619Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 12m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8070m 7696Mi 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:51 DEBUG } 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG } 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG --- status code --- 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG --- stderr --- 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG ] 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG "data": { 06:07:51 DEBUG { 06:07:51 DEBUG ] 06:07:51 DEBUG { 06:07:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:07:51 DEBUG "data": { 06:07:51 DEBUG 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG } 06:07:51 DEBUG { 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG } 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG --- http response --- 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "data": { 06:07:51 DEBUG "data": { 06:07:51 DEBUG "data": { 06:07:51 DEBUG { 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG } 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG } 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "data": { 06:07:51 DEBUG } 06:07:51 DEBUG "data": { 06:07:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG "data": { 06:07:51 DEBUG "data": { 06:07:51 DEBUG "data": { 06:07:51 DEBUG { 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG ----- output ----- 06:07:51 DEBUG { 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "data": { 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "result": [ 06:07:51 DEBUG { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "result": [ 06:07:51 DEBUG "instance": "10.142.0.107:9100" 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG "status": "success", 06:07:51 DEBUG "pod": "ds-cts-0" 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "result": [ 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG { 06:07:51 DEBUG }, 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "data": { 06:07:51 DEBUG }, 06:07:51 DEBUG "instance": "10.142.0.107:9100" 06:07:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:07:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:07:51 DEBUG { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "instance": "10.142.0.107:9100" 06:07:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:07:51 DEBUG "resultType": "vector", 06:07:51 DEBUG "value": [ 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG "metric": { 06:07:51 DEBUG "pod": "ds-idrepo-0" 06:07:51 DEBUG "pod": "ds-idrepo-0" 06:07:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:07:51 DEBUG 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG "result": [ 06:07:51 DEBUG 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG { 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG }, 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "metric": { 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "pod": "am-767f6b67b5-9zw22" 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG }, 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG [...] 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG [...] 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG [...] 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG [...] 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "instance": "10.142.0.43:9100" 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "pod": "ds-cts-2" 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG }, 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG }, 06:07:51 DEBUG "instance": "10.142.0.43:9100" 06:07:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:07:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "instance": "10.142.0.43:9100" 06:07:51 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:07:51 DEBUG [...] 06:07:51 DEBUG "value": [ 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG 06:07:51 DEBUG "pod": "ds-cts-2" 06:07:51 DEBUG "pod": "ds-idrepo-2" 06:07:51 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG }, 06:07:51 DEBUG "49031260.63333333" 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 06:07:51 DEBUG "30" 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG }, 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG "value": [ 06:07:51 DEBUG ] 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG "8169091072" 06:07:51 DEBUG "8.042004578995684" 06:07:51 DEBUG "value": [ 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG } 06:07:51 DEBUG "0.22656666666664763" 06:07:51 DEBUG "274.3745185383662" 06:07:51 DEBUG "pod": "am-767f6b67b5-vbltz" 06:07:51 DEBUG "3157995209728" 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG "0" 06:07:51 DEBUG "0" 06:07:51 DEBUG "0" 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG }, 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG "1342.1666666666667" 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG "value": [ 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG 1713589671, 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG "1342.2666666666667" 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG } 06:07:51 DEBUG ] 06:07:51 DEBUG } 06:07:51 DEBUG } 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589728" 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 DEBUG --- status code --- 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589728" 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 DEBUG --- status code --- 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:48 DEBUG ----- output ----- 06:08:48 DEBUG { 06:08:48 DEBUG "status": "success", 06:08:48 DEBUG "data": { 06:08:48 DEBUG "resultType": "vector", 06:08:48 DEBUG "result": [ 06:08:48 DEBUG { 06:08:48 DEBUG "metric": { 06:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG [...] 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 1713589728, 06:08:48 DEBUG "19.966666666666665" 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG } 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589728" 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589728" 06:08:48 ERROR [http_cmd]: ERROR 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 ERROR --- status code --- 06:08:48 DEBUG --- status code --- 06:08:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 ERROR --- http response --- 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:08:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:08:48 DEBUG ----- output ----- 06:08:48 DEBUG { 06:08:48 DEBUG "status": "success", 06:08:48 DEBUG "data": { 06:08:48 DEBUG "resultType": "vector", 06:08:48 DEBUG "result": [ 06:08:48 DEBUG { 06:08:48 DEBUG "metric": { 06:08:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG [...] 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 1713589728, 06:08:48 DEBUG "284536422.77038544" 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG } 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589728" 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 DEBUG --- status code --- 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:48 DEBUG ----- output ----- 06:08:48 DEBUG { 06:08:48 DEBUG "status": "success", 06:08:48 DEBUG "data": { 06:08:48 DEBUG "resultType": "vector", 06:08:48 DEBUG "result": [ 06:08:48 DEBUG { 06:08:48 DEBUG "metric": { 06:08:48 DEBUG "instance": "10.142.0.107:9100" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG [...] 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG "instance": "10.142.0.43:9100" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 1713589728, 06:08:48 DEBUG "33466093.233333334" 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG } 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589728" 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589728" 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 DEBUG --- status code --- 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:08:48 DEBUG ----- output ----- 06:08:48 DEBUG { 06:08:48 DEBUG "status": "success", 06:08:48 DEBUG "data": { 06:08:48 DEBUG "resultType": "vector", 06:08:48 DEBUG "result": [ 06:08:48 DEBUG { 06:08:48 DEBUG "metric": { 06:08:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG [...] 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 1713589728, 06:08:48 DEBUG "4.077420613170116" 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG } 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 DEBUG --- status code --- 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:48 DEBUG ----- output ----- 06:08:48 DEBUG { 06:08:48 DEBUG "status": "success", 06:08:48 DEBUG "data": { 06:08:48 DEBUG "resultType": "vector", 06:08:48 DEBUG "result": [ 06:08:48 DEBUG { 06:08:48 DEBUG "metric": { 06:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG [...] 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 1713589728, 06:08:48 DEBUG "0" 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG } 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589728" 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 DEBUG --- status code --- 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:48 DEBUG ----- output ----- 06:08:48 DEBUG { 06:08:48 DEBUG "status": "success", 06:08:48 DEBUG "data": { 06:08:48 DEBUG "resultType": "vector", 06:08:48 DEBUG "result": [ 06:08:48 DEBUG { 06:08:48 DEBUG "metric": { 06:08:48 DEBUG "pod": "ds-idrepo-0" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG [...] 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG "pod": "ds-idrepo-2" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 1713589728, 06:08:48 DEBUG "0" 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG } 06:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589728" 06:08:48 INFO [http_cmd]: http status code OK 06:08:48 DEBUG --- status code --- 06:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:48 DEBUG --- http response --- 06:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:48 DEBUG ----- output ----- 06:08:48 DEBUG { 06:08:48 DEBUG "status": "success", 06:08:48 DEBUG "data": { 06:08:48 DEBUG "resultType": "vector", 06:08:48 DEBUG "result": [ 06:08:48 DEBUG { 06:08:48 DEBUG "metric": { 06:08:48 DEBUG "instance": "10.142.0.107:9100" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG [...] 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG 06:08:48 DEBUG "instance": "10.142.0.43:9100" 06:08:48 DEBUG }, 06:08:48 DEBUG "value": [ 06:08:48 DEBUG 1713589728, 06:08:48 DEBUG "0.02133333333334614" 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG ] 06:08:48 DEBUG } 06:08:48 DEBUG } 06:08:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589730" 06:08:50 INFO [http_cmd]: http status code OK 06:08:50 DEBUG --- status code --- 06:08:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:50 DEBUG --- http response --- 06:08:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:50 DEBUG ----- output ----- 06:08:50 DEBUG { 06:08:50 DEBUG "status": "success", 06:08:50 DEBUG "data": { 06:08:50 DEBUG "resultType": "vector", 06:08:50 DEBUG "result": [ 06:08:50 DEBUG { 06:08:50 DEBUG "metric": { 06:08:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:08:50 DEBUG }, 06:08:50 DEBUG "value": [ 06:08:50 DEBUG 06:08:50 DEBUG 06:08:50 DEBUG 06:08:50 DEBUG [...] 06:08:50 DEBUG 06:08:50 DEBUG 06:08:50 DEBUG 06:08:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:08:50 DEBUG }, 06:08:50 DEBUG "value": [ 06:08:50 DEBUG 1713589730, 06:08:50 DEBUG "1211.4" 06:08:50 DEBUG ] 06:08:50 DEBUG } 06:08:50 DEBUG ] 06:08:50 DEBUG } 06:08:50 DEBUG } 06:08:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589731" 06:08:51 INFO [http_cmd]: http status code OK 06:08:51 DEBUG --- status code --- 06:08:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:51 DEBUG --- http response --- 06:08:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:08:51 DEBUG ----- output ----- 06:08:51 DEBUG { 06:08:51 DEBUG "status": "success", 06:08:51 DEBUG "data": { 06:08:51 DEBUG "resultType": "vector", 06:08:51 DEBUG "result": [ 06:08:51 DEBUG { 06:08:51 DEBUG "metric": { 06:08:51 DEBUG "pod": "ds-cts-0" 06:08:51 DEBUG }, 06:08:51 DEBUG "value": [ 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG [...] 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG "pod": "ds-cts-2" 06:08:51 DEBUG }, 06:08:51 DEBUG "value": [ 06:08:51 DEBUG 1713589731, 06:08:51 DEBUG "0" 06:08:51 DEBUG ] 06:08:51 DEBUG } 06:08:51 DEBUG ] 06:08:51 DEBUG } 06:08:51 DEBUG } 06:08:51 INFO 06:08:51 INFO [loop_until]: kubectl --namespace=xlou top node 06:08:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:08:51 INFO [loop_until]: OK (rc = 0) 06:08:51 DEBUG --- stdout --- 06:08:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6207Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3184m 10% 7380Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6206m 20% 11364Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3368Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3809m 12% 13800Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 5152Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6360m 21% 11602Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 7399m 24% 3694Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 6924m 23% 10861Mi 9% gke-xlou-cdm-ds-0f601751-j488 3286m 11% 6935Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3367m 11% 15716Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3251m 10% 7596Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3732m 12% 13230Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8236m 51% 11063Mi 18% 06:08:51 DEBUG --- stderr --- 06:08:51 DEBUG 06:08:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589731" 06:08:51 INFO [http_cmd]: http status code OK 06:08:51 DEBUG --- status code --- 06:08:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:51 DEBUG --- http response --- 06:08:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:51 DEBUG ----- output ----- 06:08:51 DEBUG { 06:08:51 DEBUG "status": "success", 06:08:51 DEBUG "data": { 06:08:51 DEBUG "resultType": "vector", 06:08:51 DEBUG "result": [ 06:08:51 DEBUG { 06:08:51 DEBUG "metric": { 06:08:51 DEBUG "pod": "ds-idrepo-0" 06:08:51 DEBUG }, 06:08:51 DEBUG "value": [ 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG [...] 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG 06:08:51 DEBUG "pod": "ds-idrepo-2" 06:08:51 DEBUG }, 06:08:51 DEBUG "value": [ 06:08:51 DEBUG 1713589731, 06:08:51 DEBUG "0" 06:08:51 DEBUG ] 06:08:51 DEBUG } 06:08:51 DEBUG ] 06:08:51 DEBUG } 06:08:51 DEBUG } 06:08:51 INFO 06:08:51 INFO [loop_until]: kubectl --namespace=xlou top pods 06:08:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:08:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589731" 06:08:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589731" 06:08:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589731" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589732" 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589732" 06:08:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589732" 06:08:52 DEBUG --- status code --- 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG --- status code --- 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 DEBUG --- status code --- 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG { 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:08:52 INFO [http_cmd]: http status code OK 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG --- status code --- 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG { 06:08:52 DEBUG { 06:08:52 DEBUG { 06:08:52 DEBUG { 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG { 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG { 06:08:52 DEBUG "data": { 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG { 06:08:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:08:52 DEBUG { 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG { 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG { 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG { 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG --- http response --- 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "data": { 06:08:52 DEBUG "data": { 06:08:52 DEBUG "data": { 06:08:52 DEBUG "data": { 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "data": { 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "data": { 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "data": { 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:08:52 DEBUG "data": { 06:08:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "data": { 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "data": { 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG { 06:08:52 DEBUG { 06:08:52 DEBUG "data": { 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG ----- output ----- 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "result": [ 06:08:52 DEBUG { 06:08:52 DEBUG "result": [ 06:08:52 DEBUG { 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG { 06:08:52 DEBUG { 06:08:52 DEBUG { 06:08:52 DEBUG "result": [ 06:08:52 DEBUG { 06:08:52 DEBUG "result": [ 06:08:52 DEBUG { 06:08:52 DEBUG "data": { 06:08:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:08:52 DEBUG "result": [ 06:08:52 DEBUG { 06:08:52 DEBUG "status": "success", 06:08:52 DEBUG { 06:08:52 DEBUG { 06:08:52 DEBUG "data": { 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "metric": { 06:08:52 DEBUG { 06:08:52 DEBUG "metric": { 06:08:52 DEBUG { 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG }, 06:08:52 DEBUG { 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "data": { 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "metric": { 06:08:52 INFO [loop_until]: OK (rc = 0) 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "pod": "ds-cts-0" 06:08:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:08:52 DEBUG "pod": "ds-cts-0" 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "instance": "10.142.0.107:9100" 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:08:52 DEBUG "result": [ 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "metric": { 06:08:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:08:52 DEBUG "resultType": "vector", 06:08:52 DEBUG "pod": "ds-idrepo-0" 06:08:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:08:52 DEBUG --- stdout --- 06:08:52 DEBUG "result": [ 06:08:52 DEBUG }, 06:08:52 DEBUG }, 06:08:52 DEBUG }, 06:08:52 DEBUG "pod": "ds-idrepo-0" 06:08:52 DEBUG }, 06:08:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:08:52 DEBUG }, 06:08:52 DEBUG { 06:08:52 DEBUG 06:08:52 DEBUG "instance": "10.142.0.107:9100" 06:08:52 DEBUG }, 06:08:52 DEBUG "result": [ 06:08:52 DEBUG }, 06:08:52 DEBUG }, 06:08:52 DEBUG { 06:08:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6327m 8650Mi am-767f6b67b5-qcqz2 6431m 8708Mi am-767f6b67b5-vbltz 6729m 8645Mi ds-cts-0 3565m 10650Mi ds-cts-1 3530m 10729Mi ds-cts-2 3890m 10611Mi ds-idrepo-0 3075m 3714Mi ds-idrepo-1 3158m 3706Mi ds-idrepo-2 3249m 3618Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 11m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7872m 7851Mi 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "value": [ 06:08:52 DEBUG }, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG }, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "metric": { 06:08:52 DEBUG 06:08:52 DEBUG }, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG { 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "metric": { 06:08:52 DEBUG --- stderr --- 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG "metric": { 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "instance": "10.142.0.107:9100" 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG }, 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG }, 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG }, 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG }, 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG }, 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG 06:08:52 DEBUG [...] 06:08:52 DEBUG "pod": "ds-cts-2" 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "instance": "10.142.0.43:9100" 06:08:52 DEBUG 06:08:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG "instance": "10.142.0.43:9100" 06:08:52 DEBUG 06:08:52 DEBUG 06:08:52 DEBUG "pod": "ds-cts-2" 06:08:52 DEBUG 06:08:52 DEBUG }, 06:08:52 DEBUG "pod": "ds-cts-2" 06:08:52 DEBUG 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG }, 06:08:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:08:52 DEBUG }, 06:08:52 DEBUG 06:08:52 DEBUG 1713589731, 06:08:52 DEBUG "pod": "ds-cts-0" 06:08:52 DEBUG }, 06:08:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:08:52 DEBUG 06:08:52 DEBUG }, 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG }, 06:08:52 DEBUG "pod": "ds-idrepo-2" 06:08:52 DEBUG "8.022252995372108" 06:08:52 DEBUG "value": [ 06:08:52 DEBUG }, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG "0.092" 06:08:52 DEBUG }, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG }, 06:08:52 DEBUG 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 06:08:52 DEBUG 1713589731, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG }, 06:08:52 DEBUG ] 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG "pod": "ds-cts-0" 06:08:52 DEBUG ] 06:08:52 DEBUG "value": [ 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG "instance": "10.142.0.43:9100" 06:08:52 DEBUG "0" 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG "value": [ 06:08:52 DEBUG } 06:08:52 DEBUG "33457629.8" 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG "8329482240" 06:08:52 DEBUG }, 06:08:52 DEBUG } 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG "2612.9465484824063" 06:08:52 DEBUG 1713589731, 06:08:52 DEBUG }, 06:08:52 DEBUG "0" 06:08:52 DEBUG }, 06:08:52 DEBUG ] 06:08:52 DEBUG "29.933333333333334" 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG ] 06:08:52 DEBUG ] 06:08:52 DEBUG "898.9333333333335" 06:08:52 DEBUG ] 06:08:52 DEBUG "value": [ 06:08:52 DEBUG ] 06:08:52 DEBUG "0.03276666666659101" 06:08:52 DEBUG ] 06:08:52 DEBUG "243256" 06:08:52 DEBUG "value": [ 06:08:52 DEBUG ] 06:08:52 DEBUG "value": [ 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG "0" 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG } 06:08:52 DEBUG 1713589732, 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG "21662977.16894977" 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG "898.2666666666667" 06:08:52 DEBUG ] 06:08:52 DEBUG "3159920866304" 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG ] 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:08:52 DEBUG } 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING No values available for selected timeframe. 06:09:07 WARNING File /tmp/lodemon_data-20240420_053236/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [20/Apr/2024 06:09:07] "GET /monitoring/average?start_time=20240420_050657&stop_time=20240420_050706 HTTP/1.1" 200 - 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589788" 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589788" 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589788" 06:09:48 ERROR [http_cmd]: ERROR 06:09:48 ERROR --- status code --- 06:09:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:09:48 ERROR --- http response --- 06:09:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:09:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:48 DEBUG ----- output ----- 06:09:48 DEBUG { 06:09:48 DEBUG "status": "success", 06:09:48 DEBUG "data": { 06:09:48 DEBUG "resultType": "vector", 06:09:48 DEBUG "result": [ 06:09:48 DEBUG { 06:09:48 DEBUG "metric": { 06:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG [...] 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 1713589788, 06:09:48 DEBUG "1750.3666666666666" 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG } 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589788" 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:09:48 DEBUG ----- output ----- 06:09:48 DEBUG { 06:09:48 DEBUG "status": "success", 06:09:48 DEBUG "data": { 06:09:48 DEBUG "resultType": "vector", 06:09:48 DEBUG "result": [ 06:09:48 DEBUG { 06:09:48 DEBUG "metric": { 06:09:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG [...] 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG "pod": "ds-cts-2" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 1713589788, 06:09:48 DEBUG "289336634.11100864" 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG } 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589788" 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:48 DEBUG ----- output ----- 06:09:48 DEBUG { 06:09:48 DEBUG "status": "success", 06:09:48 DEBUG "data": { 06:09:48 DEBUG "resultType": "vector", 06:09:48 DEBUG "result": [ 06:09:48 DEBUG { 06:09:48 DEBUG "metric": { 06:09:48 DEBUG "instance": "10.142.0.107:9100" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG [...] 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG "instance": "10.142.0.43:9100" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 1713589788, 06:09:48 DEBUG "9190958.433333334" 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG } 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589788" 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 06:09:48 DEBUG ----- output ----- 06:09:48 DEBUG { 06:09:48 DEBUG "status": "success", 06:09:48 DEBUG "data": { 06:09:48 DEBUG "resultType": "vector", 06:09:48 DEBUG "result": [ 06:09:48 DEBUG { 06:09:48 DEBUG "metric": { 06:09:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG [...] 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG "pod": "ds-cts-2" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 1713589788, 06:09:48 DEBUG "3.85855119230343" 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG } 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589788" 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:48 DEBUG ----- output ----- 06:09:48 DEBUG { 06:09:48 DEBUG "status": "success", 06:09:48 DEBUG "data": { 06:09:48 DEBUG "resultType": "vector", 06:09:48 DEBUG "result": [ 06:09:48 DEBUG { 06:09:48 DEBUG "metric": { 06:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG [...] 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 1713589788, 06:09:48 DEBUG "0" 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG } 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589788" 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:48 DEBUG ----- output ----- 06:09:48 DEBUG { 06:09:48 DEBUG "status": "success", 06:09:48 DEBUG "data": { 06:09:48 DEBUG "resultType": "vector", 06:09:48 DEBUG "result": [ 06:09:48 DEBUG { 06:09:48 DEBUG "metric": { 06:09:48 DEBUG "instance": "10.142.0.107:9100" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG [...] 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG "instance": "10.142.0.43:9100" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 1713589788, 06:09:48 DEBUG "0.011333333333334394" 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG } 06:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589788" 06:09:48 INFO [http_cmd]: http status code OK 06:09:48 DEBUG --- status code --- 06:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:48 DEBUG --- http response --- 06:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:48 DEBUG ----- output ----- 06:09:48 DEBUG { 06:09:48 DEBUG "status": "success", 06:09:48 DEBUG "data": { 06:09:48 DEBUG "resultType": "vector", 06:09:48 DEBUG "result": [ 06:09:48 DEBUG { 06:09:48 DEBUG "metric": { 06:09:48 DEBUG "pod": "ds-idrepo-0" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG [...] 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG 06:09:48 DEBUG "pod": "ds-idrepo-2" 06:09:48 DEBUG }, 06:09:48 DEBUG "value": [ 06:09:48 DEBUG 1713589788, 06:09:48 DEBUG "0" 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG ] 06:09:48 DEBUG } 06:09:48 DEBUG } 06:09:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589790" 06:09:50 INFO [http_cmd]: http status code OK 06:09:50 DEBUG --- status code --- 06:09:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:50 DEBUG --- http response --- 06:09:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:50 DEBUG ----- output ----- 06:09:50 DEBUG { 06:09:50 DEBUG "status": "success", 06:09:50 DEBUG "data": { 06:09:50 DEBUG "resultType": "vector", 06:09:50 DEBUG "result": [ 06:09:50 DEBUG { 06:09:50 DEBUG "metric": { 06:09:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:09:50 DEBUG }, 06:09:50 DEBUG "value": [ 06:09:50 DEBUG 06:09:50 DEBUG 06:09:50 DEBUG 06:09:50 DEBUG [...] 06:09:50 DEBUG 06:09:50 DEBUG 06:09:50 DEBUG 06:09:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:50 DEBUG }, 06:09:50 DEBUG "value": [ 06:09:50 DEBUG 1713589790, 06:09:50 DEBUG "0" 06:09:50 DEBUG ] 06:09:50 DEBUG } 06:09:50 DEBUG ] 06:09:50 DEBUG } 06:09:50 DEBUG } 06:09:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589791" 06:09:51 INFO [http_cmd]: http status code OK 06:09:51 DEBUG --- status code --- 06:09:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:51 DEBUG --- http response --- 06:09:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:09:51 DEBUG ----- output ----- 06:09:51 DEBUG { 06:09:51 DEBUG "status": "success", 06:09:51 DEBUG "data": { 06:09:51 DEBUG "resultType": "vector", 06:09:51 DEBUG "result": [ 06:09:51 DEBUG { 06:09:51 DEBUG "metric": { 06:09:51 DEBUG "pod": "ds-cts-0" 06:09:51 DEBUG }, 06:09:51 DEBUG "value": [ 06:09:51 DEBUG 06:09:51 DEBUG 06:09:51 DEBUG 06:09:51 DEBUG [...] 06:09:51 DEBUG 06:09:51 DEBUG 06:09:51 DEBUG 06:09:51 DEBUG "pod": "ds-cts-2" 06:09:51 DEBUG }, 06:09:51 DEBUG "value": [ 06:09:51 DEBUG 1713589791, 06:09:51 DEBUG "0" 06:09:51 DEBUG ] 06:09:51 DEBUG } 06:09:51 DEBUG ] 06:09:51 DEBUG } 06:09:51 DEBUG } 06:09:51 INFO 06:09:51 INFO [loop_until]: kubectl --namespace=xlou top node 06:09:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "data": { 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "result": [ 06:09:52 DEBUG { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "pod": "ds-idrepo-0" 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG [...] 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "pod": "ds-idrepo-2" 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG "0" 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 INFO [loop_until]: OK (rc = 0) 06:09:52 DEBUG --- stdout --- 06:09:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 6204Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 601m 2% 7385Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1615m 5% 11404Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3393Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 4471m 14% 13787Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5184Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1703m 5% 11615Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 83m 0% 3407Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 2096m 7% 10890Mi 9% gke-xlou-cdm-ds-0f601751-j488 536m 1% 6932Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 4672m 15% 15723Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 599m 2% 7612Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4907m 16% 13218Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 84m 0% 3901Mi 6% 06:09:52 DEBUG --- stderr --- 06:09:52 DEBUG 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589792" 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "data": { 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "result": [ 06:09:52 DEBUG { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG [...] 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG "114446336" 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- http response --- 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589792" 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "data": { 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "result": [ 06:09:52 DEBUG { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "pod": "ds-cts-0" 06:09:52 DEBUG }, 06:09:52 INFO 06:09:52 DEBUG "value": [ 06:09:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:09:52 DEBUG 06:09:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG [...] 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "pod": "ds-cts-2" 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG "30.03333333333333" 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "data": { 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG "result": [ 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "data": { 06:09:52 DEBUG "instance": "10.142.0.107:9100" 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG }, 06:09:52 DEBUG "result": [ 06:09:52 DEBUG "value": [ 06:09:52 DEBUG { 06:09:52 DEBUG 06:09:52 DEBUG "metric": { 06:09:52 DEBUG 06:09:52 DEBUG "instance": "10.142.0.107:9100" 06:09:52 DEBUG 06:09:52 DEBUG }, 06:09:52 DEBUG [...] 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "instance": "10.142.0.43:9100" 06:09:52 DEBUG [...] 06:09:52 DEBUG }, 06:09:52 DEBUG 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG 06:09:52 DEBUG "0.015799999999991127" 06:09:52 DEBUG "instance": "10.142.0.43:9100" 06:09:52 DEBUG ] 06:09:52 DEBUG }, 06:09:52 DEBUG } 06:09:52 DEBUG "value": [ 06:09:52 DEBUG ] 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG } 06:09:52 DEBUG "8926680.566666666" 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589792" 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG --- status code --- 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG --- http response --- 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:09:52 DEBUG ----- output ----- 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG --- status code --- 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG { 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG { 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "data": { 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG "data": { 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:09:52 DEBUG "data": { 06:09:52 DEBUG "data": { 06:09:52 INFO [http_cmd]: http status code OK 06:09:52 DEBUG "data": { 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG "data": { 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG --- status code --- 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG { 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "result": [ 06:09:52 DEBUG "result": [ 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG { 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG "result": [ 06:09:52 DEBUG "result": [ 06:09:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:09:52 DEBUG "result": [ 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "result": [ 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG --- http response --- 06:09:52 DEBUG { 06:09:52 DEBUG "data": { 06:09:52 DEBUG { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "data": { 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:09:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:09:52 DEBUG "data": { 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "data": { 06:09:52 DEBUG "pod": "ds-cts-0" 06:09:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:09:52 DEBUG "instance": "10.142.0.107:9100" 06:09:52 DEBUG ----- output ----- 06:09:52 DEBUG "result": [ 06:09:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:09:52 DEBUG }, 06:09:52 DEBUG }, 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "result": [ 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG }, 06:09:52 DEBUG }, 06:09:52 DEBUG }, 06:09:52 DEBUG { 06:09:52 DEBUG { 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "result": [ 06:09:52 DEBUG { 06:09:52 DEBUG "result": [ 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "status": "success", 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:09:52 DEBUG 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG { 06:09:52 DEBUG "metric": { 06:09:52 DEBUG { 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "data": { 06:09:52 DEBUG }, 06:09:52 INFO [loop_until]: OK (rc = 0) 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "metric": { 06:09:52 DEBUG "pod": "ds-idrepo-0" 06:09:52 DEBUG "metric": { 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "resultType": "vector", 06:09:52 DEBUG "value": [ 06:09:52 DEBUG --- stdout --- 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "pod": "ds-idrepo-0" 06:09:52 DEBUG }, 06:09:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "result": [ 06:09:52 DEBUG 06:09:52 DEBUG [...] 06:09:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1862m 8652Mi am-767f6b67b5-qcqz2 2639m 8701Mi am-767f6b67b5-vbltz 2114m 8647Mi ds-cts-0 4424m 10627Mi ds-cts-1 4985m 10725Mi ds-cts-2 5071m 10602Mi ds-idrepo-0 521m 3714Mi ds-idrepo-1 510m 3716Mi ds-idrepo-2 470m 3618Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1177Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 12m 63Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 4m 16Mi 06:09:52 DEBUG 06:09:52 DEBUG [...] 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG }, 06:09:52 DEBUG [...] 06:09:52 DEBUG [...] 06:09:52 DEBUG [...] 06:09:52 DEBUG { 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG --- stderr --- 06:09:52 DEBUG [...] 06:09:52 DEBUG 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "metric": { 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:09:52 DEBUG [...] 06:09:52 DEBUG 06:09:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG }, 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG }, 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG [...] 06:09:52 DEBUG 06:09:52 DEBUG "pod": "ds-cts-2" 06:09:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:52 DEBUG "instance": "10.142.0.43:9100" 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:52 DEBUG [...] 06:09:52 DEBUG 06:09:52 DEBUG [...] 06:09:52 DEBUG }, 06:09:52 DEBUG }, 06:09:52 DEBUG }, 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG }, 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG }, 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "0.092" 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG 06:09:52 DEBUG }, 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG ] 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG 06:09:52 DEBUG "pod": "ds-idrepo-2" 06:09:52 DEBUG 06:09:52 DEBUG "0" 06:09:52 DEBUG "0" 06:09:52 DEBUG "3161265296384" 06:09:52 DEBUG [...] 06:09:52 DEBUG "value": [ 06:09:52 DEBUG "0.03512052036339096" 06:09:52 DEBUG } 06:09:52 DEBUG "0" 06:09:52 DEBUG "pod": "ds-cts-2" 06:09:52 DEBUG }, 06:09:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:52 DEBUG ] 06:09:52 DEBUG ] 06:09:52 DEBUG ] 06:09:52 DEBUG 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG ] 06:09:52 DEBUG ] 06:09:52 DEBUG ] 06:09:52 DEBUG }, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG }, 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG 06:09:52 DEBUG "0.10000000000000002" 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG "value": [ 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG "value": [ 06:09:52 DEBUG ] 06:09:52 DEBUG ] 06:09:52 DEBUG ] 06:09:52 DEBUG 06:09:52 DEBUG ] 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG "0" 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG "0" 06:09:52 DEBUG ] 06:09:52 DEBUG "0.03333333333333333" 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG }, 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG "value": [ 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG 1713589792, 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG "164587" 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:09:52 DEBUG ] 06:09:52 DEBUG } 06:09:52 DEBUG } 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589848" 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589848" 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589848" 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589848" 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 DEBUG --- status code --- 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:10:48 ERROR [http_cmd]: ERROR 06:10:48 ERROR --- status code --- 06:10:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:10:48 ERROR --- http response --- 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:10:48 DEBUG --- status code --- 06:10:48 DEBUG --- status code --- 06:10:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:48 DEBUG ----- output ----- 06:10:48 DEBUG ----- output ----- 06:10:48 DEBUG { 06:10:48 DEBUG { 06:10:48 DEBUG "status": "success", 06:10:48 DEBUG "status": "success", 06:10:48 DEBUG "data": { 06:10:48 DEBUG "data": { 06:10:48 DEBUG "resultType": "vector", 06:10:48 DEBUG "resultType": "vector", 06:10:48 DEBUG "result": [ 06:10:48 DEBUG "result": [ 06:10:48 DEBUG { 06:10:48 DEBUG { 06:10:48 DEBUG "metric": { 06:10:48 DEBUG "metric": { 06:10:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:10:48 DEBUG }, 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG [...] 06:10:48 DEBUG [...] 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:10:48 DEBUG }, 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 1713589848, 06:10:48 DEBUG 1713589848, 06:10:48 DEBUG "22352935.968989056" 06:10:48 DEBUG "1341.4666666666667" 06:10:48 DEBUG ] 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 DEBUG ] 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589848" 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 DEBUG --- status code --- 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:48 DEBUG ----- output ----- 06:10:48 DEBUG { 06:10:48 DEBUG "status": "success", 06:10:48 DEBUG "data": { 06:10:48 DEBUG "resultType": "vector", 06:10:48 DEBUG "result": [ 06:10:48 DEBUG { 06:10:48 DEBUG "metric": { 06:10:48 DEBUG "instance": "10.142.0.107:9100" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG [...] 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG "instance": "10.142.0.43:9100" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 1713589848, 06:10:48 DEBUG "8832899.666666666" 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589848" 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 DEBUG --- status code --- 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:10:48 DEBUG ----- output ----- 06:10:48 DEBUG { 06:10:48 DEBUG "status": "success", 06:10:48 DEBUG "data": { 06:10:48 DEBUG "resultType": "vector", 06:10:48 DEBUG "result": [ 06:10:48 DEBUG { 06:10:48 DEBUG "metric": { 06:10:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG [...] 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 1713589848, 06:10:48 DEBUG "2.130717722768356" 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589848" 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 DEBUG --- status code --- 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:48 DEBUG ----- output ----- 06:10:48 DEBUG { 06:10:48 DEBUG "status": "success", 06:10:48 DEBUG "data": { 06:10:48 DEBUG "resultType": "vector", 06:10:48 DEBUG "result": [ 06:10:48 DEBUG { 06:10:48 DEBUG "metric": { 06:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG [...] 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 1713589848, 06:10:48 DEBUG "0" 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589848" 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 DEBUG --- status code --- 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:48 DEBUG ----- output ----- 06:10:48 DEBUG { 06:10:48 DEBUG "status": "success", 06:10:48 DEBUG "data": { 06:10:48 DEBUG "resultType": "vector", 06:10:48 DEBUG "result": [ 06:10:48 DEBUG { 06:10:48 DEBUG "metric": { 06:10:48 DEBUG "instance": "10.142.0.107:9100" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG [...] 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG "instance": "10.142.0.43:9100" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 1713589848, 06:10:48 DEBUG "0.009333333333340002" 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589848" 06:10:48 INFO [http_cmd]: http status code OK 06:10:48 DEBUG --- status code --- 06:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:48 DEBUG --- http response --- 06:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:48 DEBUG ----- output ----- 06:10:48 DEBUG { 06:10:48 DEBUG "status": "success", 06:10:48 DEBUG "data": { 06:10:48 DEBUG "resultType": "vector", 06:10:48 DEBUG "result": [ 06:10:48 DEBUG { 06:10:48 DEBUG "metric": { 06:10:48 DEBUG "pod": "ds-idrepo-0" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG [...] 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG 06:10:48 DEBUG "pod": "ds-idrepo-2" 06:10:48 DEBUG }, 06:10:48 DEBUG "value": [ 06:10:48 DEBUG 1713589848, 06:10:48 DEBUG "0.03333333333333333" 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG ] 06:10:48 DEBUG } 06:10:48 DEBUG } 06:10:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589850" 06:10:50 INFO [http_cmd]: http status code OK 06:10:50 DEBUG --- status code --- 06:10:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:50 DEBUG --- http response --- 06:10:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:50 DEBUG ----- output ----- 06:10:50 DEBUG { 06:10:50 DEBUG "status": "success", 06:10:50 DEBUG "data": { 06:10:50 DEBUG "resultType": "vector", 06:10:50 DEBUG "result": [ 06:10:50 DEBUG { 06:10:50 DEBUG "metric": { 06:10:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:10:50 DEBUG }, 06:10:50 DEBUG "value": [ 06:10:50 DEBUG 06:10:50 DEBUG 06:10:50 DEBUG 06:10:50 DEBUG [...] 06:10:50 DEBUG 06:10:50 DEBUG 06:10:50 DEBUG 06:10:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:10:50 DEBUG }, 06:10:50 DEBUG "value": [ 06:10:50 DEBUG 1713589850, 06:10:50 DEBUG "0" 06:10:50 DEBUG ] 06:10:50 DEBUG } 06:10:50 DEBUG ] 06:10:50 DEBUG } 06:10:50 DEBUG } 06:10:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589851" 06:10:51 INFO [http_cmd]: http status code OK 06:10:51 DEBUG --- status code --- 06:10:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:51 DEBUG --- http response --- 06:10:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:10:51 DEBUG ----- output ----- 06:10:51 DEBUG { 06:10:51 DEBUG "status": "success", 06:10:51 DEBUG "data": { 06:10:51 DEBUG "resultType": "vector", 06:10:51 DEBUG "result": [ 06:10:51 DEBUG { 06:10:51 DEBUG "metric": { 06:10:51 DEBUG "pod": "ds-cts-0" 06:10:51 DEBUG }, 06:10:51 DEBUG "value": [ 06:10:51 DEBUG 06:10:51 DEBUG 06:10:51 DEBUG 06:10:51 DEBUG [...] 06:10:51 DEBUG 06:10:51 DEBUG 06:10:51 DEBUG 06:10:51 DEBUG "pod": "ds-cts-2" 06:10:51 DEBUG }, 06:10:51 DEBUG "value": [ 06:10:51 DEBUG 1713589851, 06:10:51 DEBUG "0.000032104444444444446" 06:10:51 DEBUG ] 06:10:51 DEBUG } 06:10:51 DEBUG ] 06:10:51 DEBUG } 06:10:51 DEBUG } 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "pod": "ds-idrepo-0" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "pod": "ds-idrepo-2" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG "0.03333333333333333" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 INFO 06:10:52 INFO [loop_until]: kubectl --namespace=xlou top node 06:10:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589852" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "pod": "ds-cts-0" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "pod": "ds-cts-2" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG "30" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "instance": "10.142.0.107:9100" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "instance": "10.142.0.43:9100" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG "0.015199999999956996" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589852" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG "621531136" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589852" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "instance": "10.142.0.107:9100" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "instance": "10.142.0.43:9100" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 INFO [loop_until]: OK (rc = 0) 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG --- stdout --- 06:10:52 DEBUG "8824001.766666666" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6210Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1111m 3% 7350Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1285m 4% 11342Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3382Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3956m 13% 13767Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5146Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1384m 4% 11569Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 82m 0% 3348Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1596m 5% 10847Mi 9% gke-xlou-cdm-ds-0f601751-j488 819m 2% 6973Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3537m 11% 15729Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 891m 2% 7581Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3957m 13% 13566Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 1135m 7% 4410Mi 7% 06:10:52 DEBUG --- stderr --- 06:10:52 DEBUG 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589852" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG "83644.5" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG { 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "pod": "ds-cts-0" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "pod": "ds-cts-2" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG "0" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "instance": "10.142.0.107:9100" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "instance": "10.142.0.43:9100" 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG "3161478202368" 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589852" 06:10:52 INFO 06:10:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:10:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589852" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:52 DEBUG { 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG { 06:10:52 DEBUG { 06:10:52 DEBUG "data": { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "data": { 06:10:52 DEBUG "data": { 06:10:52 DEBUG "result": [ 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG { 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG "result": [ 06:10:52 DEBUG "result": [ 06:10:52 DEBUG "metric": { 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG { 06:10:52 DEBUG { 06:10:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "metric": { 06:10:52 DEBUG }, 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:10:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG "value": [ 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG }, 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG }, 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG { 06:10:52 DEBUG "value": [ 06:10:52 DEBUG { 06:10:52 DEBUG "value": [ 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG 06:10:52 INFO [http_cmd]: http status code OK 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG 06:10:52 DEBUG --- status code --- 06:10:52 DEBUG "data": { 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 06:10:52 DEBUG 06:10:52 DEBUG "data": { 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:10:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG --- http response --- 06:10:52 DEBUG "result": [ 06:10:52 DEBUG { 06:10:52 DEBUG [...] 06:10:52 DEBUG "result": [ 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG { 06:10:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:10:52 DEBUG { 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG 06:10:52 DEBUG { 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG ----- output ----- 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "data": { 06:10:52 DEBUG 06:10:52 DEBUG "metric": { 06:10:52 DEBUG 06:10:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:10:52 DEBUG "data": { 06:10:52 DEBUG { 06:10:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG 06:10:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:10:52 DEBUG 06:10:52 DEBUG }, 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG "status": "success", 06:10:52 DEBUG }, 06:10:52 DEBUG "result": [ 06:10:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:10:52 DEBUG }, 06:10:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:10:52 DEBUG "value": [ 06:10:52 DEBUG "result": [ 06:10:52 DEBUG "data": { 06:10:52 DEBUG "value": [ 06:10:52 DEBUG { 06:10:52 DEBUG }, 06:10:52 DEBUG "value": [ 06:10:52 DEBUG }, 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG { 06:10:52 DEBUG "resultType": "vector", 06:10:52 DEBUG 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG "value": [ 06:10:52 DEBUG "1.0898824641281497" 06:10:52 DEBUG "metric": { 06:10:52 DEBUG "result": [ 06:10:52 DEBUG 06:10:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG ] 06:10:52 DEBUG "pod": "ds-idrepo-0" 06:10:52 DEBUG { 06:10:52 DEBUG 06:10:52 DEBUG }, 06:10:52 DEBUG "0" 06:10:52 DEBUG 06:10:52 DEBUG "0" 06:10:52 DEBUG } 06:10:52 DEBUG }, 06:10:52 DEBUG "metric": { 06:10:52 DEBUG [...] 06:10:52 DEBUG "value": [ 06:10:52 DEBUG ] 06:10:52 DEBUG [...] 06:10:52 DEBUG ] 06:10:52 DEBUG ] 06:10:52 DEBUG "value": [ 06:10:52 DEBUG "pod": "ds-idrepo-0" 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG } 06:10:52 DEBUG 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG 06:10:52 DEBUG }, 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG ] 06:10:52 DEBUG 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG } 06:10:52 DEBUG 06:10:52 DEBUG } 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:10:52 DEBUG [...] 06:10:52 DEBUG } 06:10:52 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:10:52 DEBUG } 06:10:52 DEBUG [...] 06:10:52 DEBUG 06:10:52 DEBUG }, 06:10:52 DEBUG 06:10:52 DEBUG }, 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG "value": [ 06:10:52 DEBUG 06:10:52 DEBUG [...] 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG 06:10:52 DEBUG 06:10:52 DEBUG "0.092" 06:10:52 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 06:10:52 DEBUG "0" 06:10:52 DEBUG "pod": "ds-idrepo-2" 06:10:52 DEBUG 06:10:52 DEBUG ] 06:10:52 DEBUG }, 06:10:52 DEBUG ] 06:10:52 DEBUG }, 06:10:52 DEBUG 06:10:52 DEBUG } 06:10:52 DEBUG "value": [ 06:10:52 DEBUG } 06:10:52 DEBUG "value": [ 06:10:52 DEBUG "pod": "ds-cts-2" 06:10:52 DEBUG ] 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG ] 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG }, 06:10:52 DEBUG } 06:10:52 DEBUG "24868306.60052862" 06:10:52 DEBUG } 06:10:52 DEBUG "0.03333333333333333" 06:10:52 DEBUG "value": [ 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG 1713589852, 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG "0" 06:10:52 DEBUG ] 06:10:52 DEBUG ] 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 DEBUG ] 06:10:52 DEBUG } 06:10:52 DEBUG } 06:10:52 INFO [loop_until]: OK (rc = 0) 06:10:52 DEBUG --- stdout --- 06:10:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1285m 8605Mi am-767f6b67b5-qcqz2 1331m 8674Mi am-767f6b67b5-vbltz 1505m 8619Mi ds-cts-0 4009m 10625Mi ds-cts-1 3499m 10700Mi ds-cts-2 3974m 10743Mi ds-idrepo-0 1187m 3719Mi ds-idrepo-1 812m 3731Mi ds-idrepo-2 814m 3686Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1178Mi idm-b85f9d978-wnhxb 4m 1236Mi lodemon-799cdc5bdb-7t5x7 11m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1040m 503Mi 06:10:52 DEBUG --- stderr --- 06:10:52 DEBUG 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589908" 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589908" 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589908" 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589908" 06:11:48 ERROR [http_cmd]: ERROR 06:11:48 ERROR --- status code --- 06:11:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:11:48 ERROR --- http response --- 06:11:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:11:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:48 DEBUG ----- output ----- 06:11:48 DEBUG { 06:11:48 DEBUG "status": "success", 06:11:48 DEBUG "data": { 06:11:48 DEBUG "resultType": "vector", 06:11:48 DEBUG "result": [ 06:11:48 DEBUG { 06:11:48 DEBUG "metric": { 06:11:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG [...] 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG "pod": "ds-cts-0" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 1713589908, 06:11:48 DEBUG "50709.54801673755" 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG } 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:48 DEBUG ----- output ----- 06:11:48 DEBUG { 06:11:48 DEBUG "status": "success", 06:11:48 DEBUG "data": { 06:11:48 DEBUG "resultType": "vector", 06:11:48 DEBUG "result": [ 06:11:48 DEBUG { 06:11:48 DEBUG "metric": { 06:11:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG [...] 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 1713589908, 06:11:48 DEBUG "1213.7333333333333" 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG } 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589908" 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:48 DEBUG ----- output ----- 06:11:48 DEBUG { 06:11:48 DEBUG "status": "success", 06:11:48 DEBUG "data": { 06:11:48 DEBUG "resultType": "vector", 06:11:48 DEBUG "result": [ 06:11:48 DEBUG { 06:11:48 DEBUG "metric": { 06:11:48 DEBUG "instance": "10.142.0.107:9100" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG [...] 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG "instance": "10.142.0.43:9100" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 1713589908, 06:11:48 DEBUG "8025309.3" 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG } 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589908" 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:48 DEBUG ----- output ----- 06:11:48 DEBUG { 06:11:48 DEBUG "status": "success", 06:11:48 DEBUG "data": { 06:11:48 DEBUG "resultType": "vector", 06:11:48 DEBUG "result": [ 06:11:48 DEBUG { 06:11:48 DEBUG "metric": { 06:11:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG [...] 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG "pod": "ds-cts-0" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 1713589908, 06:11:48 DEBUG "3.403887886019499" 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG } 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589908" 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:48 DEBUG ----- output ----- 06:11:48 DEBUG { 06:11:48 DEBUG "status": "success", 06:11:48 DEBUG "data": { 06:11:48 DEBUG "resultType": "vector", 06:11:48 DEBUG "result": [ 06:11:48 DEBUG { 06:11:48 DEBUG "metric": { 06:11:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG [...] 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 1713589908, 06:11:48 DEBUG "0" 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG } 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589908" 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:48 DEBUG ----- output ----- 06:11:48 DEBUG { 06:11:48 DEBUG "status": "success", 06:11:48 DEBUG "data": { 06:11:48 DEBUG "resultType": "vector", 06:11:48 DEBUG "result": [ 06:11:48 DEBUG { 06:11:48 DEBUG "metric": { 06:11:48 DEBUG "instance": "10.142.0.107:9100" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG [...] 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG "instance": "10.142.0.43:9100" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 1713589908, 06:11:48 DEBUG "0.010333333333327725" 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG } 06:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589908" 06:11:48 INFO [http_cmd]: http status code OK 06:11:48 DEBUG --- status code --- 06:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:48 DEBUG --- http response --- 06:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:48 DEBUG ----- output ----- 06:11:48 DEBUG { 06:11:48 DEBUG "status": "success", 06:11:48 DEBUG "data": { 06:11:48 DEBUG "resultType": "vector", 06:11:48 DEBUG "result": [ 06:11:48 DEBUG { 06:11:48 DEBUG "metric": { 06:11:48 DEBUG "pod": "ds-idrepo-0" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG [...] 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG 06:11:48 DEBUG "pod": "ds-idrepo-2" 06:11:48 DEBUG }, 06:11:48 DEBUG "value": [ 06:11:48 DEBUG 1713589908, 06:11:48 DEBUG "0" 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG ] 06:11:48 DEBUG } 06:11:48 DEBUG } 06:11:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589910" 06:11:50 INFO [http_cmd]: http status code OK 06:11:50 DEBUG --- status code --- 06:11:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:50 DEBUG --- http response --- 06:11:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:50 DEBUG ----- output ----- 06:11:50 DEBUG { 06:11:50 DEBUG "status": "success", 06:11:50 DEBUG "data": { 06:11:50 DEBUG "resultType": "vector", 06:11:50 DEBUG "result": [ 06:11:50 DEBUG { 06:11:50 DEBUG "metric": { 06:11:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:11:50 DEBUG }, 06:11:50 DEBUG "value": [ 06:11:50 DEBUG 06:11:50 DEBUG 06:11:50 DEBUG 06:11:50 DEBUG [...] 06:11:50 DEBUG 06:11:50 DEBUG 06:11:50 DEBUG 06:11:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:11:50 DEBUG }, 06:11:50 DEBUG "value": [ 06:11:50 DEBUG 1713589910, 06:11:50 DEBUG "0" 06:11:50 DEBUG ] 06:11:50 DEBUG } 06:11:50 DEBUG ] 06:11:50 DEBUG } 06:11:50 DEBUG } 06:11:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589911" 06:11:51 INFO [http_cmd]: http status code OK 06:11:51 DEBUG --- status code --- 06:11:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:51 DEBUG --- http response --- 06:11:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:11:51 DEBUG ----- output ----- 06:11:51 DEBUG { 06:11:51 DEBUG "status": "success", 06:11:51 DEBUG "data": { 06:11:51 DEBUG "resultType": "vector", 06:11:51 DEBUG "result": [ 06:11:51 DEBUG { 06:11:51 DEBUG "metric": { 06:11:51 DEBUG "pod": "ds-cts-0" 06:11:51 DEBUG }, 06:11:51 DEBUG "value": [ 06:11:51 DEBUG 06:11:51 DEBUG 06:11:51 DEBUG 06:11:51 DEBUG [...] 06:11:51 DEBUG 06:11:51 DEBUG 06:11:51 DEBUG 06:11:51 DEBUG "pod": "ds-cts-2" 06:11:51 DEBUG }, 06:11:51 DEBUG "value": [ 06:11:51 DEBUG 1713589911, 06:11:51 DEBUG "0.000032104444444444446" 06:11:51 DEBUG ] 06:11:51 DEBUG } 06:11:51 DEBUG ] 06:11:51 DEBUG } 06:11:51 DEBUG } 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "pod": "ds-idrepo-0" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "ds-idrepo-2" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "0" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "pod": "ds-cts-0" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "ds-cts-2" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "30" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "instance": "10.142.0.107:9100" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "instance": "10.142.0.43:9100" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "0.015300000000010527" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "13567" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713589912" 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "pod": "ds-cts-0" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "ds-cts-2" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "0" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "1783717888" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO 06:11:52 INFO [loop_until]: kubectl --namespace=xlou top node 06:11:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589912" 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "instance": "10.142.0.107:9100" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "instance": "10.142.0.43:9100" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "8016501.566666666" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589912" 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "instance": "10.142.0.107:9100" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "instance": "10.142.0.43:9100" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "3161700713472" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO 06:11:52 INFO [loop_until]: kubectl --namespace=xlou top pods 06:11:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713589912" 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "ds-cts-0" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "2832.9356433931594" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG ----- output ----- 06:11:52 INFO [loop_until]: OK (rc = 0) 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG { 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- stdout --- 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG "status": "success", 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6201Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 417m 1% 7371Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1211m 4% 11355Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3379Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3436m 11% 13772Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5168Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1297m 4% 11570Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3296Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1447m 4% 10851Mi 9% gke-xlou-cdm-ds-0f601751-j488 407m 1% 7012Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3152m 10% 15713Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 461m 1% 7609Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3531m 11% 13120Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1079m 6% 5392Mi 9% 06:11:52 DEBUG "data": { 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- stderr --- 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "data": { 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG { 06:11:52 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG }, 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "data": { 06:11:52 DEBUG "value": [ 06:11:52 DEBUG { 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "data": { 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG "result": [ 06:11:52 DEBUG 06:11:52 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG { 06:11:52 DEBUG 06:11:52 DEBUG }, 06:11:52 DEBUG "result": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG [...] 06:11:52 DEBUG "value": [ 06:11:52 DEBUG { 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:11:52 DEBUG "pod": "ds-idrepo-0" 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "metric": { 06:11:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:11:52 DEBUG }, 06:11:52 DEBUG }, 06:11:52 INFO [http_cmd]: http status code OK 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "am-767f6b67b5-9zw22" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG "value": [ 06:11:52 DEBUG --- status code --- 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:11:52 DEBUG "pod": "ds-cts-0" 06:11:52 DEBUG [...] 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG --- http response --- 06:11:52 DEBUG }, 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG [...] 06:11:52 DEBUG ----- output ----- 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG { 06:11:52 DEBUG "2556299.4778158176" 06:11:52 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 06:11:52 DEBUG [...] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "status": "success", 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "data": { 06:11:52 DEBUG }, 06:11:52 DEBUG ] 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:11:52 DEBUG "pod": "ds-idrepo-2" 06:11:52 DEBUG "resultType": "vector", 06:11:52 DEBUG "value": [ 06:11:52 DEBUG } 06:11:52 DEBUG 06:11:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:11:52 DEBUG }, 06:11:52 DEBUG }, 06:11:52 DEBUG "result": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG } 06:11:52 DEBUG "pod": "am-767f6b67b5-vbltz" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG "value": [ 06:11:52 DEBUG { 06:11:52 DEBUG "1.0341132214332185" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "metric": { 06:11:52 DEBUG ] 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "0" 06:11:52 DEBUG "0" 06:11:52 DEBUG "pod": "ds-idrepo-0" 06:11:52 DEBUG } 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "0" 06:11:52 DEBUG ] 06:11:52 DEBUG ] 06:11:52 DEBUG }, 06:11:52 DEBUG ] 06:11:52 DEBUG "0.092" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 DEBUG "value": [ 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG ] 06:11:52 DEBUG 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 DEBUG 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 DEBUG 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 DEBUG [...] 06:11:52 DEBUG } 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG 06:11:52 DEBUG "pod": "ds-cts-2" 06:11:52 DEBUG }, 06:11:52 DEBUG "value": [ 06:11:52 DEBUG 1713589912, 06:11:52 DEBUG "0" 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG ] 06:11:52 DEBUG } 06:11:52 DEBUG } 06:11:52 INFO [loop_until]: OK (rc = 0) 06:11:52 DEBUG --- stdout --- 06:11:52 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1231m 8606Mi am-767f6b67b5-qcqz2 1272m 8665Mi am-767f6b67b5-vbltz 1366m 8618Mi ds-cts-0 3452m 10632Mi ds-cts-1 3135m 10697Mi ds-cts-2 3603m 10503Mi ds-idrepo-0 395m 3720Mi ds-idrepo-1 391m 3731Mi ds-idrepo-2 370m 3688Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1236Mi lodemon-799cdc5bdb-7t5x7 12m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 978m 1345Mi 06:11:52 DEBUG --- stderr --- 06:11:52 DEBUG 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713589968" 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589968" 06:12:48 ERROR [http_cmd]: ERROR 06:12:48 ERROR --- status code --- 06:12:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 06:12:48 ERROR --- http response --- 06:12:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 06:12:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589968" 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:12:48 DEBUG ----- output ----- 06:12:48 DEBUG { 06:12:48 DEBUG "status": "success", 06:12:48 DEBUG "data": { 06:12:48 DEBUG "resultType": "vector", 06:12:48 DEBUG "result": [ 06:12:48 DEBUG { 06:12:48 DEBUG "metric": { 06:12:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG [...] 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 1713589968, 06:12:48 DEBUG "316406567.335645" 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG } 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589968" 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:48 DEBUG ----- output ----- 06:12:48 DEBUG { 06:12:48 DEBUG "status": "success", 06:12:48 DEBUG "data": { 06:12:48 DEBUG "resultType": "vector", 06:12:48 DEBUG "result": [ 06:12:48 DEBUG { 06:12:48 DEBUG "metric": { 06:12:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG [...] 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 1713589968, 06:12:48 DEBUG "0" 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG } 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589968" 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:48 DEBUG ----- output ----- 06:12:48 DEBUG { 06:12:48 DEBUG "status": "success", 06:12:48 DEBUG "data": { 06:12:48 DEBUG "resultType": "vector", 06:12:48 DEBUG "result": [ 06:12:48 DEBUG { 06:12:48 DEBUG "metric": { 06:12:48 DEBUG "instance": "10.142.0.107:9100" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG [...] 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG "instance": "10.142.0.43:9100" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 1713589968, 06:12:48 DEBUG "34678.333333333336" 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG } 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589968" 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589968" 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:48 DEBUG ----- output ----- 06:12:48 DEBUG { 06:12:48 DEBUG "status": "success", 06:12:48 DEBUG "data": { 06:12:48 DEBUG "resultType": "vector", 06:12:48 DEBUG "result": [ 06:12:48 DEBUG { 06:12:48 DEBUG "metric": { 06:12:48 DEBUG "pod": "am-767f6b67b5-9zw22" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG [...] 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 1713589968, 06:12:48 DEBUG "0" 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG } 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 06:12:48 DEBUG ----- output ----- 06:12:48 DEBUG { 06:12:48 DEBUG "status": "success", 06:12:48 DEBUG "data": { 06:12:48 DEBUG "resultType": "vector", 06:12:48 DEBUG "result": [ 06:12:48 DEBUG { 06:12:48 DEBUG "metric": { 06:12:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG [...] 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 1713589968, 06:12:48 DEBUG "0.005811950543038185" 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG } 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713589968" 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 06:12:48 DEBUG ----- output ----- 06:12:48 DEBUG { 06:12:48 DEBUG "status": "success", 06:12:48 DEBUG "data": { 06:12:48 DEBUG "resultType": "vector", 06:12:48 DEBUG "result": [ 06:12:48 DEBUG { 06:12:48 DEBUG "metric": { 06:12:48 DEBUG "instance": "10.142.0.107:9100" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG [...] 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG "instance": "10.142.0.43:9100" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 1713589968, 06:12:48 DEBUG "0.0013333333333321207" 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG } 06:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589968" 06:12:48 INFO [http_cmd]: http status code OK 06:12:48 DEBUG --- status code --- 06:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:48 DEBUG --- http response --- 06:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:48 DEBUG ----- output ----- 06:12:48 DEBUG { 06:12:48 DEBUG "status": "success", 06:12:48 DEBUG "data": { 06:12:48 DEBUG "resultType": "vector", 06:12:48 DEBUG "result": [ 06:12:48 DEBUG { 06:12:48 DEBUG "metric": { 06:12:48 DEBUG "pod": "ds-idrepo-0" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG [...] 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG 06:12:48 DEBUG "pod": "ds-idrepo-2" 06:12:48 DEBUG }, 06:12:48 DEBUG "value": [ 06:12:48 DEBUG 1713589968, 06:12:48 DEBUG "0" 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG ] 06:12:48 DEBUG } 06:12:48 DEBUG } 06:12:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713589970" 06:12:50 INFO [http_cmd]: http status code OK 06:12:50 DEBUG --- status code --- 06:12:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 06:12:50 DEBUG --- http response --- 06:12:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 06:12:50 DEBUG ----- output ----- 06:12:50 DEBUG { 06:12:50 DEBUG "status": "success", 06:12:50 DEBUG "data": { 06:12:50 DEBUG "resultType": "vector", 06:12:50 DEBUG "result": [ 06:12:50 DEBUG { 06:12:50 DEBUG "metric": { 06:12:50 DEBUG "pod": "am-767f6b67b5-9zw22" 06:12:50 DEBUG }, 06:12:50 DEBUG "value": [ 06:12:50 DEBUG 06:12:50 DEBUG 06:12:50 DEBUG 06:12:50 DEBUG [...] 06:12:50 DEBUG 06:12:50 DEBUG 06:12:50 DEBUG 06:12:50 DEBUG "pod": "am-767f6b67b5-vbltz" 06:12:50 DEBUG }, 06:12:50 DEBUG "value": [ 06:12:50 DEBUG 1713589970, 06:12:50 DEBUG "0" 06:12:50 DEBUG ] 06:12:50 DEBUG } 06:12:50 DEBUG ] 06:12:50 DEBUG } 06:12:50 DEBUG } 06:12:51 INFO Finished: True 06:12:51 INFO Waiting for threads to register finish flag 06:13:50 INFO Done. Have a nice day! :) 127.0.0.1 - - [20/Apr/2024 06:13:50] "GET /monitoring/stop HTTP/1.1" 200 - 06:13:53 WARNING File /tmp/lodemon_data-20240420_053236/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [20/Apr/2024 06:13:57] "GET /monitoring/process HTTP/1.1" 200 -