==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-597977695c-ld9s4 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Sat, 20 Apr 2024 19:40:34 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=597977695c skaffold.dev/run-id=1aab4c14-33b5-4c0e-8177-d7a3d4a891fc Annotations: Status: Running IP: 10.106.42.192 IPs: IP: 10.106.42.192 Controlled By: ReplicaSet/lodemon-597977695c Containers: lodemon: Container ID: containerd://f806359d126c1246dcc8f313780a926cb8e5b0eba16834e328ae6f1ca4a369b7 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sat, 20 Apr 2024 19:40:35 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qlwbw (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-qlwbw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 44m default-scheduler Successfully assigned xlou/lodemon-597977695c-ld9s4 to gke-xlou-cdm-default-pool-1b0769c8-7vmj Normal Pulling 44m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 44m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 280.566126ms (280.572399ms including waiting) Normal Created 44m kubelet Created container lodemon Normal Started 44m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.42.192:8080 Press CTRL+C to quit 20:41:43 INFO 20:41:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:43 INFO [loop_until]: OK (rc = 0) 20:41:43 DEBUG --- stdout --- 20:41:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:43 DEBUG --- stderr --- 20:41:43 DEBUG 20:41:43 INFO 20:41:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:43 INFO [loop_until]: OK (rc = 0) 20:41:43 DEBUG --- stdout --- 20:41:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:43 DEBUG --- stderr --- 20:41:43 DEBUG 20:41:43 INFO 20:41:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:43 INFO [loop_until]: OK (rc = 0) 20:41:43 DEBUG --- stdout --- 20:41:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:43 DEBUG --- stderr --- 20:41:43 DEBUG 20:41:43 INFO 20:41:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:43 INFO [loop_until]: OK (rc = 0) 20:41:43 DEBUG --- stdout --- 20:41:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:43 DEBUG --- stderr --- 20:41:43 DEBUG 20:41:43 INFO 20:41:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:43 INFO [loop_until]: OK (rc = 0) 20:41:43 DEBUG --- stdout --- 20:41:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:43 DEBUG --- stderr --- 20:41:43 DEBUG 20:41:43 INFO 20:41:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:43 INFO [loop_until]: OK (rc = 0) 20:41:43 DEBUG --- stdout --- 20:41:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:43 DEBUG --- stderr --- 20:41:43 DEBUG 20:41:43 INFO 20:41:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:44 INFO 20:41:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:44 INFO 20:41:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:44 INFO 20:41:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:44 INFO 20:41:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:44 INFO 20:41:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:44 INFO 20:41:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:44 INFO 20:41:44 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:44 INFO [loop_until]: OK (rc = 0) 20:41:44 DEBUG --- stdout --- 20:41:44 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:44 DEBUG --- stderr --- 20:41:44 DEBUG 20:41:45 INFO 20:41:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:45 INFO [loop_until]: OK (rc = 0) 20:41:45 DEBUG --- stdout --- 20:41:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:45 DEBUG --- stderr --- 20:41:45 DEBUG 20:41:45 INFO 20:41:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:45 INFO [loop_until]: OK (rc = 0) 20:41:45 DEBUG --- stdout --- 20:41:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:45 DEBUG --- stderr --- 20:41:45 DEBUG 20:41:45 INFO 20:41:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:45 INFO [loop_until]: OK (rc = 0) 20:41:45 DEBUG --- stdout --- 20:41:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:45 DEBUG --- stderr --- 20:41:45 DEBUG 20:41:45 INFO 20:41:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:45 INFO [loop_until]: OK (rc = 0) 20:41:45 DEBUG --- stdout --- 20:41:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:45 DEBUG --- stderr --- 20:41:45 DEBUG 20:41:45 INFO 20:41:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:45 INFO [loop_until]: OK (rc = 0) 20:41:45 DEBUG --- stdout --- 20:41:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:45 DEBUG --- stderr --- 20:41:45 DEBUG 20:41:45 INFO 20:41:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:45 INFO [loop_until]: OK (rc = 0) 20:41:45 DEBUG --- stdout --- 20:41:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:45 DEBUG --- stderr --- 20:41:45 DEBUG 20:41:45 INFO 20:41:45 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:45 INFO [loop_until]: OK (rc = 0) 20:41:45 DEBUG --- stdout --- 20:41:45 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:45 DEBUG --- stderr --- 20:41:45 DEBUG 20:41:46 INFO 20:41:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:46 DEBUG --- stderr --- 20:41:46 DEBUG 20:41:46 INFO 20:41:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:46 DEBUG --- stderr --- 20:41:46 DEBUG 20:41:46 INFO 20:41:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:46 DEBUG --- stderr --- 20:41:46 DEBUG 20:41:46 INFO 20:41:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:46 DEBUG --- stderr --- 20:41:46 DEBUG 20:41:46 INFO 20:41:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:46 DEBUG --- stderr --- 20:41:46 DEBUG 20:41:46 INFO 20:41:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:41:46 DEBUG --- stderr --- 20:41:46 DEBUG 20:41:46 INFO Initializing monitoring instance threads 20:41:46 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 20:41:46 INFO Starting instance threads 20:41:46 INFO 20:41:46 INFO Thread started 20:41:46 INFO [loop_until]: kubectl --namespace=xlou top node 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO 20:41:46 INFO [loop_until]: kubectl --namespace=xlou top pods 20:41:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG --- status code --- 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 DEBUG --- status code --- 20:41:46 INFO Thread started 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG --- http response --- 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG --- http response --- 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG --- status code --- 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG { 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG ----- output ----- 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG { 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG ----- output ----- 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG { 20:41:46 DEBUG { 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG { 20:41:46 DEBUG { 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG { 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 DEBUG { 20:41:46 DEBUG "data": { 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG "data": { 20:41:46 DEBUG { 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG { 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "status": "success", 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG "status": "success", 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 INFO Thread started 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG "data": { 20:41:46 DEBUG "data": { 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG { 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG { 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG "data": { 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG "data": { 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG "data": { 20:41:46 DEBUG "data": { 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 DEBUG "result": [ 20:41:46 INFO Thread started 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "data": { 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "data": { 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8602Mi am-767f6b67b5-qcqz2 7m 9018Mi am-767f6b67b5-vbltz 7m 8610Mi ds-cts-0 13m 10991Mi ds-cts-1 12m 10700Mi ds-cts-2 13m 11061Mi ds-idrepo-0 14m 3900Mi ds-idrepo-1 11m 3891Mi ds-idrepo-2 13m 3837Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-597977695c-ld9s4 1m 54Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 1m 3848Mi 20:41:46 DEBUG { 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG { 20:41:46 DEBUG "result": [ 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 DEBUG "result": [ 20:41:46 INFO Thread started 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG "data": { 20:41:46 DEBUG { 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG "data": { 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG "result": [ 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG "result": [ 20:41:46 DEBUG --- stderr --- 20:41:46 INFO [loop_until]: OK (rc = 0) 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "result": [ 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "metric": { 20:41:46 DEBUG { 20:41:46 DEBUG { 20:41:46 DEBUG { 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 DEBUG "resultType": "vector", 20:41:46 INFO Thread started 20:41:46 DEBUG "metric": { 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG "result": [ 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG "result": [ 20:41:46 DEBUG { 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG { 20:41:46 DEBUG 20:41:46 DEBUG --- stdout --- 20:41:46 DEBUG "data": { 20:41:46 DEBUG { 20:41:46 DEBUG { 20:41:46 DEBUG { 20:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG "result": [ 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642106" 20:41:46 DEBUG "instance": "10.142.0.107:9100" 20:41:46 INFO Thread started 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG { 20:41:46 DEBUG "result": [ 20:41:46 DEBUG { 20:41:46 DEBUG "metric": { 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG "metric": { 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 7637Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 58m 0% 11311Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3375Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 14207Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5202Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 11906Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2930Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 10810Mi 9% gke-xlou-cdm-ds-0f601751-j488 64m 0% 7206Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 15803Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 7842Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 13713Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 9404Mi 16% 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "metric": { 20:41:46 DEBUG }, 20:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:46 DEBUG "data": { 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG { 20:41:46 DEBUG }, 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- http response --- 20:41:46 INFO Thread started 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG "metric": { 20:41:46 DEBUG { 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:41:46 DEBUG { 20:41:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG "result": [ 20:41:46 DEBUG --- stderr --- 20:41:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:41:46 DEBUG "data": { 20:41:46 DEBUG "instance": "10.142.0.107:9100" 20:41:46 DEBUG "value": [ 20:41:46 DEBUG }, 20:41:46 DEBUG }, 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "value": [ 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642106" 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:46 DEBUG --- http response --- 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642106" 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:46 DEBUG }, 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "instance": "10.142.0.107:9100" 20:41:46 DEBUG }, 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG }, 20:41:46 DEBUG { 20:41:46 DEBUG { 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG }, 20:41:46 DEBUG 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "pod": "ds-cts-0" 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:46 DEBUG 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG { 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "data": { 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "value": [ 20:41:46 ERROR [http_cmd]: ERROR 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG { 20:41:46 DEBUG }, 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG 20:41:46 DEBUG { 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:46 DEBUG ----- output ----- 20:41:46 INFO Thread started 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "pod": "ds-idrepo-0" 20:41:46 DEBUG 20:41:46 ERROR --- status code --- 20:41:46 DEBUG { 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "value": [ 20:41:46 DEBUG { 20:41:46 DEBUG 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG { 20:41:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642106" 20:41:46 DEBUG "data": { 20:41:46 INFO Thread started 20:41:46 DEBUG 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "data": { 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG 20:41:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:41:46 DEBUG "metric": { 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "pod": "ds-idrepo-0" 20:41:46 DEBUG 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG [...] 20:41:46 DEBUG "data": { 20:41:46 DEBUG { 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "resultType": "vector", 20:41:46 INFO All threads has been started 127.0.0.1 - - [20/Apr/2024 20:41:46] "GET /monitoring/start HTTP/1.1" 200 - 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG { 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG [...] 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 ERROR --- http response --- 20:41:46 DEBUG "pod": "ds-idrepo-0" 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:46 DEBUG [...] 20:41:46 DEBUG }, 20:41:46 DEBUG 20:41:46 DEBUG "data": { 20:41:46 DEBUG 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG "data": { 20:41:46 DEBUG "result": [ 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG [...] 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "result": [ 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG }, 20:41:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:41:46 DEBUG [...] 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "data": { 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG { 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG 20:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:46 DEBUG { 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG { 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG "result": [ 20:41:46 DEBUG 20:41:46 DEBUG { 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG "result": [ 20:41:46 DEBUG { 20:41:46 DEBUG --- http response --- 20:41:46 INFO [http_cmd]: http status code OK 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG "metric": { 20:41:46 DEBUG 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG { 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "instance": "10.142.0.43:9100" 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "metric": { 20:41:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:41:46 DEBUG --- status code --- 20:41:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG [...] 20:41:46 DEBUG 20:41:46 DEBUG "instance": "10.142.0.43:9100" 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG "data": { 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "pod": "ds-cts-0" 20:41:46 DEBUG }, 20:41:46 DEBUG { 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:46 DEBUG 20:41:46 DEBUG --- http response --- 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG }, 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "instance": "10.142.0.43:9100" 20:41:46 DEBUG "value": [ 20:41:46 DEBUG }, 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG }, 20:41:46 DEBUG [...] 20:41:46 DEBUG 20:41:46 DEBUG "pod": "ds-cts-2" 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:46 DEBUG }, 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG { 20:41:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:46 DEBUG 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:41:46 DEBUG "pod": "ds-idrepo-2" 20:41:46 DEBUG }, 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG "pod": "ds-cts-0" 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG }, 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG }, 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG ----- output ----- 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG 20:41:46 DEBUG "33162.23333333333" 20:41:46 DEBUG 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "0.006458191266112521" 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG { 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG "28793.966666666667" 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "data": { 20:41:46 DEBUG { 20:41:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG "293222657.70417273" 20:41:46 DEBUG [...] 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG ] 20:41:46 DEBUG "399.0258158792011" 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "0" 20:41:46 DEBUG "pod": "ds-idrepo-2" 20:41:46 DEBUG }, 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG "status": "success", 20:41:46 DEBUG }, 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG [...] 20:41:46 DEBUG } 20:41:46 DEBUG [...] 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG "0.0009999999999990906" 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG "pod": "ds-idrepo-0" 20:41:46 DEBUG ] 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "0" 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "0" 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "data": { 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "0" 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG "0.024573205208943853" 20:41:46 DEBUG "0" 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG }, 20:41:46 DEBUG } 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG [...] 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG ] 20:41:46 DEBUG { 20:41:46 DEBUG "resultType": "vector", 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG "value": [ 20:41:46 DEBUG ] 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG [...] 20:41:46 DEBUG [...] 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG "metric": { 20:41:46 DEBUG "result": [ 20:41:46 DEBUG "4131377152" 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG "pod": "ds-idrepo-2" 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG "0" 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG "instance": "10.142.0.107:9100" 20:41:46 DEBUG { 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG } 20:41:46 DEBUG }, 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG [...] 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG }, 20:41:46 DEBUG "metric": { 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG }, 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG "pod": "ds-cts-2" 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG } 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "instance": "10.142.0.107:9100" 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG [...] 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:46 DEBUG }, 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG "0" 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG "value": [ 20:41:46 DEBUG }, 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "0" 20:41:46 DEBUG "0" 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG "pod": "ds-cts-2" 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "1" 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG }, 20:41:46 DEBUG "0" 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG "pod": "ds-cts-2" 20:41:46 DEBUG "value": [ 20:41:46 DEBUG ] 20:41:46 DEBUG "0" 20:41:46 DEBUG "0.16" 20:41:46 DEBUG } 20:41:46 DEBUG [...] 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG }, 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG 20:41:46 DEBUG [...] 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG "value": [ 20:41:46 DEBUG "0" 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG 20:41:46 DEBUG "0" 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG "instance": "10.142.0.43:9100" 20:41:46 DEBUG "instance": "10.142.0.43:9100" 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG }, 20:41:46 DEBUG ] 20:41:46 DEBUG }, 20:41:46 DEBUG "value": [ 20:41:46 DEBUG } 20:41:46 DEBUG "value": [ 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG } 20:41:46 DEBUG 1713642106, 20:41:46 DEBUG "0.0015333333333207597" 20:41:46 DEBUG "3764120165376" 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG ] 20:41:46 DEBUG ] 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:41:46 DEBUG } 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO 20:42:46 INFO [loop_until]: kubectl --namespace=xlou top pods 20:42:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO 20:42:46 INFO [loop_until]: kubectl --namespace=xlou top node 20:42:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG { 20:42:46 DEBUG "status": "success", 20:42:46 DEBUG "data": { 20:42:46 DEBUG "resultType": "vector", 20:42:46 DEBUG "result": [ 20:42:46 DEBUG { 20:42:46 DEBUG "metric": { 20:42:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:46 DEBUG }, 20:42:46 DEBUG "value": [ 20:42:46 DEBUG 20:42:46 DEBUG 20:42:46 DEBUG 20:42:46 DEBUG [...] 20:42:46 DEBUG 20:42:46 DEBUG 20:42:46 DEBUG 20:42:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:46 DEBUG }, 20:42:46 DEBUG "value": [ 20:42:46 DEBUG 1713642166, 20:42:46 DEBUG "24998.58356286817" 20:42:46 DEBUG ] 20:42:46 DEBUG } 20:42:46 DEBUG ] 20:42:46 DEBUG } 20:42:46 DEBUG } 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- http response --- 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642166" 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG --- status code --- 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG { 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG "status": "success", 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG "data": { 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG "resultType": "vector", 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG "result": [ 20:42:46 DEBUG { 20:42:46 DEBUG { 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG { 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642166" 20:42:46 DEBUG "status": "success", 20:42:46 DEBUG "status": "success", 20:42:46 DEBUG { 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG "metric": { 20:42:46 DEBUG "data": { 20:42:46 DEBUG "data": { 20:42:46 DEBUG "status": "success", 20:42:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:42:46 DEBUG { 20:42:46 DEBUG "pod": "ds-cts-0" 20:42:46 DEBUG "resultType": "vector", 20:42:46 DEBUG "resultType": "vector", 20:42:46 DEBUG "data": { 20:42:46 DEBUG "status": "success", 20:42:46 DEBUG }, 20:42:46 DEBUG "result": [ 20:42:46 DEBUG "result": [ 20:42:46 DEBUG "resultType": "vector", 20:42:46 DEBUG "data": { 20:42:46 DEBUG "value": [ 20:42:46 DEBUG { 20:42:46 DEBUG { 20:42:46 DEBUG "result": [ 20:42:46 DEBUG "resultType": "vector", 20:42:46 ERROR [http_cmd]: ERROR 20:42:46 DEBUG 20:42:46 DEBUG "metric": { 20:42:46 DEBUG "metric": { 20:42:46 DEBUG { 20:42:46 DEBUG "result": [ 20:42:46 ERROR --- status code --- 20:42:46 DEBUG 20:42:46 DEBUG "instance": "10.142.0.107:9100" 20:42:46 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642166" 20:42:46 DEBUG "metric": { 20:42:46 DEBUG { 20:42:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:42:46 DEBUG 20:42:46 DEBUG }, 20:42:46 DEBUG }, 20:42:46 DEBUG "pod": "ds-idrepo-0" 20:42:46 DEBUG "metric": { 20:42:46 ERROR --- http response --- 20:42:46 DEBUG [...] 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642166" 20:42:46 DEBUG "value": [ 20:42:46 DEBUG "value": [ 20:42:46 DEBUG }, 20:42:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:42:46 DEBUG 20:42:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 DEBUG 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 DEBUG 20:42:46 DEBUG "value": [ 20:42:46 DEBUG }, 20:42:46 DEBUG 20:42:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:42:46 DEBUG 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 DEBUG --- status code --- 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642166" 20:42:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642166" 20:42:46 DEBUG 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG "value": [ 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG 20:42:46 INFO [loop_until]: OK (rc = 0) 20:42:46 DEBUG 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- status code --- 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG 20:42:46 DEBUG --- status code --- 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG --- status code --- 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG "pod": "ds-cts-2" 20:42:46 DEBUG --- stdout --- 20:42:46 DEBUG [...] 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG }, 20:42:46 DEBUG 20:42:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8603Mi am-767f6b67b5-qcqz2 439m 9019Mi am-767f6b67b5-vbltz 760m 8612Mi ds-cts-0 956m 10998Mi ds-cts-1 250m 10704Mi ds-cts-2 267m 11065Mi ds-idrepo-0 15m 3900Mi ds-idrepo-1 291m 3892Mi ds-idrepo-2 782m 3838Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-597977695c-ld9s4 129m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 3684m 4306Mi 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG [...] 20:42:46 DEBUG 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG [...] 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG --- status code --- 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG "value": [ 20:42:46 DEBUG 20:42:46 DEBUG --- stderr --- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 INFO [http_cmd]: http status code OK 20:42:46 DEBUG 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG 1713642166, 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG 20:42:46 DEBUG 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG --- status code --- 20:42:46 DEBUG 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG ----- output ----- 20:42:46 DEBUG --- http response --- 20:42:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:46 DEBUG [...] 20:42:47 DEBUG { 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG "0" 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG "instance": "10.142.0.43:9100" 20:42:47 DEBUG { 20:42:47 INFO [loop_until]: OK (rc = 0) 20:42:47 DEBUG { 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG { 20:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:47 DEBUG { 20:42:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:47 DEBUG 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG --- http response --- 20:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:47 DEBUG 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG ] 20:42:47 DEBUG { 20:42:47 DEBUG }, 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG --- stdout --- 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG }, 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG --- http response --- 20:42:47 DEBUG "pod": "ds-idrepo-2" 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG { 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG 20:42:47 DEBUG "data": { 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG } 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "data": { 20:42:47 DEBUG "data": { 20:42:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6261Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1377m 4% 7658Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 564m 1% 11330Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3373Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1013m 3% 14212Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5177Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1518m 5% 11907Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 1606m 5% 2952Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2579m 8% 10829Mi 9% gke-xlou-cdm-ds-0f601751-j488 619m 2% 7252Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 1062m 3% 15693Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 422m 1% 7852Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 423m 1% 13711Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 2877m 18% 9862Mi 16% 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "data": { 20:42:47 DEBUG { 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "data": { 20:42:47 DEBUG }, 20:42:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:47 DEBUG "data": { 20:42:47 DEBUG "data": { 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "data": { 20:42:47 DEBUG "data": { 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG { 20:42:47 DEBUG 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "data": { 20:42:47 DEBUG "data": { 20:42:47 DEBUG ] 20:42:47 DEBUG "data": { 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG --- stderr --- 20:42:47 DEBUG "data": { 20:42:47 DEBUG "data": { 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "value": [ 20:42:47 DEBUG ----- output ----- 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "data": { 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG { 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG } 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "0.020999999999992264" 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "result": [ 20:42:47 DEBUG 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "data": { 20:42:47 DEBUG "262.15468692725517" 20:42:47 DEBUG "result": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG { 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG "data": { 20:42:47 DEBUG }, 20:42:47 DEBUG { 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "result": [ 20:42:47 DEBUG } 20:42:47 DEBUG "result": [ 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG "result": [ 20:42:47 DEBUG "result": [ 20:42:47 DEBUG { 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG ] 20:42:47 DEBUG { 20:42:47 DEBUG "0" 20:42:47 DEBUG "status": "success", 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG "result": [ 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "metric": { 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG ] 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG { 20:42:47 DEBUG { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "result": [ 20:42:47 DEBUG } 20:42:47 DEBUG "metric": { 20:42:47 DEBUG ] 20:42:47 DEBUG "data": { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "data": { 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG "result": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG } 20:42:47 DEBUG "instance": "10.142.0.107:9100" 20:42:47 DEBUG "pod": "ds-idrepo-0" 20:42:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:47 DEBUG { 20:42:47 DEBUG ] 20:42:47 DEBUG "pod": "ds-idrepo-0" 20:42:47 DEBUG } 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG "instance": "10.142.0.107:9100" 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:47 DEBUG "resultType": "vector", 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "4613648384" 20:42:47 DEBUG }, 20:42:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG ] 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG "instance": "10.142.0.107:9100" 20:42:47 DEBUG }, 20:42:47 DEBUG "metric": { 20:42:47 DEBUG } 20:42:47 DEBUG }, 20:42:47 DEBUG ] 20:42:47 DEBUG "result": [ 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG "pod": "ds-cts-0" 20:42:47 DEBUG }, 20:42:47 DEBUG "result": [ 20:42:47 DEBUG { 20:42:47 DEBUG 20:42:47 DEBUG ] 20:42:47 DEBUG "value": [ 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG } 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "value": [ 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG } 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG { 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG { 20:42:47 DEBUG "metric": { 20:42:47 DEBUG 20:42:47 DEBUG } 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG } 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG }, 20:42:47 DEBUG } 20:42:47 DEBUG 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG "metric": { 20:42:47 DEBUG "pod": "ds-cts-0" 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG } 20:42:47 DEBUG 20:42:47 DEBUG "instance": "10.142.0.107:9100" 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "pod": "ds-idrepo-0" 20:42:47 DEBUG ] 20:42:47 DEBUG "value": [ 20:42:47 DEBUG [...] 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG }, 20:42:47 DEBUG } 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG [...] 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG [...] 20:42:47 DEBUG } 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "pod": "ds-idrepo-2" 20:42:47 DEBUG 20:42:47 DEBUG [...] 20:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:47 DEBUG 20:42:47 DEBUG "pod": "ds-cts-2" 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "instance": "10.142.0.43:9100" 20:42:47 DEBUG "pod": "ds-idrepo-2" 20:42:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "instance": "10.142.0.43:9100" 20:42:47 DEBUG 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:47 DEBUG 20:42:47 DEBUG "value": [ 20:42:47 DEBUG [...] 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG }, 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG "pod": "ds-cts-2" 20:42:47 DEBUG }, 20:42:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "value": [ 20:42:47 DEBUG }, 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG 20:42:47 DEBUG "instance": "10.142.0.43:9100" 20:42:47 DEBUG 20:42:47 DEBUG "0.000029326666666666665" 20:42:47 DEBUG "value": [ 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG }, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "3764510176256" 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "instance": "10.142.0.43:9100" 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG }, 20:42:47 DEBUG 20:42:47 DEBUG ] 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "0" 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "0.16" 20:42:47 DEBUG "0" 20:42:47 DEBUG "3.7854369072956366" 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "0" 20:42:47 DEBUG ] 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG }, 20:42:47 DEBUG "0" 20:42:47 DEBUG "value": [ 20:42:47 DEBUG 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "0" 20:42:47 DEBUG ] 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "38139394.86666667" 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG "262.22133359992" 20:42:47 DEBUG "0.13666376587290513" 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG "3923.5" 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG "pod": "ds-cts-2" 20:42:47 DEBUG }, 20:42:47 DEBUG "261.7881302275984" 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG "value": [ 20:42:47 DEBUG } 20:42:47 DEBUG "0.03143333333344363" 20:42:47 DEBUG } 20:42:47 DEBUG "value": [ 20:42:47 DEBUG "448.40508632947393" 20:42:47 DEBUG } 20:42:47 DEBUG "0" 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG 1713642166, 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG "38143346.766666666" 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG "0" 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG ] 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:42:47 DEBUG } 20:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642226" 20:43:46 INFO [http_cmd]: http status code OK 20:43:46 DEBUG --- status code --- 20:43:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:46 DEBUG --- http response --- 20:43:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:43:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642226" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642227" 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 1713642226, 20:43:47 DEBUG "20740.60434682537" 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 ERROR [http_cmd]: ERROR 20:43:47 ERROR --- status code --- 20:43:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:43:47 ERROR --- http response --- 20:43:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:43:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO 20:43:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:43:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG "4759511040" 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "data": { 20:43:47 DEBUG "data": { 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG "data": { 20:43:47 DEBUG "data": { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "data": { 20:43:47 DEBUG "data": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "pod": "ds-cts-0" 20:43:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:43:47 DEBUG "instance": "10.142.0.107:9100" 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:47 DEBUG "pod": "ds-cts-0" 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "result": [ 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG [...] 20:43:47 DEBUG [...] 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG [...] 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG { 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG 20:43:47 DEBUG "pod": "ds-cts-2" 20:43:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:43:47 DEBUG "instance": "10.142.0.43:9100" 20:43:47 DEBUG [...] 20:43:47 DEBUG [...] 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG "data": { 20:43:47 DEBUG "pod": "ds-cts-2" 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG }, 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG { 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG "0.00022584222222222217" 20:43:47 DEBUG "11.386740024828935" 20:43:47 DEBUG "0.042333333333346" 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG "0" 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "0" 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG ] 20:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 INFO 20:43:47 DEBUG } 20:43:47 DEBUG }, 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG 1713642227, 20:43:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:43:47 DEBUG ] 20:43:47 DEBUG "value": [ 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG "19.004824667433937" 20:43:47 DEBUG "3336.233333333333" 20:43:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:43:47 DEBUG } 20:43:47 DEBUG 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG 20:43:47 DEBUG } 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG 20:43:47 DEBUG ] 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 DEBUG ] 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 DEBUG [...] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG "3335.9666666666667" 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG { 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "pod": "ds-cts-0" 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:47 DEBUG }, 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642227" 20:43:47 DEBUG "pod": "ds-idrepo-0" 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG --- status code --- 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642227" 20:43:47 DEBUG 20:43:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642227" 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 INFO [loop_until]: OK (rc = 0) 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG "pod": "ds-cts-2" 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- stdout --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG }, 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG "pod": "ds-idrepo-2" 20:43:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17564m 8620Mi am-767f6b67b5-qcqz2 17105m 8817Mi am-767f6b67b5-vbltz 18948m 8392Mi ds-cts-0 9886m 11054Mi ds-cts-1 9064m 10764Mi ds-cts-2 9920m 11124Mi ds-idrepo-0 8177m 3901Mi ds-idrepo-1 8334m 3893Mi ds-idrepo-2 8394m 3838Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-597977695c-ld9s4 1m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11273m 4444Mi 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG }, 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG "value": [ 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG }, 20:43:47 DEBUG --- stderr --- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG "value": [ 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG { 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG "0" 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 INFO [http_cmd]: http status code OK 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG "3335.8333333333335" 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:47 DEBUG ] 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG --- status code --- 20:43:47 DEBUG "0" 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG "data": { 20:43:47 DEBUG { 20:43:47 DEBUG ] 20:43:47 DEBUG { 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG } 20:43:47 DEBUG { 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG { 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:47 DEBUG ] 20:43:47 DEBUG { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG } 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG { 20:43:47 DEBUG ] 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG { 20:43:47 DEBUG --- http response --- 20:43:47 DEBUG } 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "data": { 20:43:47 DEBUG ] 20:43:47 DEBUG "data": { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG } 20:43:47 DEBUG "data": { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG ] 20:43:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:47 DEBUG "data": { 20:43:47 DEBUG { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG } 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "data": { 20:43:47 DEBUG } 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "data": { 20:43:47 DEBUG "data": { 20:43:47 DEBUG "data": { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "data": { 20:43:47 DEBUG } 20:43:47 DEBUG ----- output ----- 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "result": [ 20:43:47 DEBUG } 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG } 20:43:47 DEBUG { 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "result": [ 20:43:47 DEBUG { 20:43:47 DEBUG "result": [ 20:43:47 DEBUG "status": "success", 20:43:47 DEBUG { 20:43:47 DEBUG }, 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG { 20:43:47 DEBUG "data": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "instance": "10.142.0.107:9100" 20:43:47 DEBUG "pod": "ds-idrepo-0" 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:47 DEBUG "metric": { 20:43:47 DEBUG "resultType": "vector", 20:43:47 DEBUG "pod": "ds-idrepo-0" 20:43:47 DEBUG 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG "pod": "ds-idrepo-0" 20:43:47 DEBUG }, 20:43:47 DEBUG "instance": "10.142.0.107:9100" 20:43:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:47 DEBUG "instance": "10.142.0.107:9100" 20:43:47 DEBUG }, 20:43:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:47 DEBUG "result": [ 20:43:47 DEBUG }, 20:43:47 DEBUG 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG }, 20:43:47 DEBUG { 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "metric": { 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "instance": "10.142.0.107:9100" 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG }, 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG [...] 20:43:47 DEBUG [...] 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG [...] 20:43:47 DEBUG 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG }, 20:43:47 DEBUG 20:43:47 DEBUG "instance": "10.142.0.43:9100" 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG }, 20:43:47 DEBUG 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG 20:43:47 DEBUG }, 20:43:47 DEBUG 20:43:47 DEBUG "pod": "ds-idrepo-2" 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG "pod": "ds-idrepo-2" 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "pod": "ds-cts-2" 20:43:47 DEBUG }, 20:43:47 DEBUG "instance": "10.142.0.43:9100" 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG "instance": "10.142.0.43:9100" 20:43:47 DEBUG [...] 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:47 DEBUG }, 20:43:47 DEBUG "139623.5" 20:43:47 DEBUG }, 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG }, 20:43:47 DEBUG 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG }, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG ] 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "3767542989824" 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 20:43:47 DEBUG "0" 20:43:47 DEBUG "value": [ 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG } 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG ] 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG "0.16" 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG 20:43:47 DEBUG ] 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG "0" 20:43:47 DEBUG ] 20:43:47 DEBUG "0" 20:43:47 DEBUG } 20:43:47 DEBUG "0" 20:43:47 DEBUG ] 20:43:47 DEBUG "79704398.96666667" 20:43:47 DEBUG "1614.4522910165874" 20:43:47 DEBUG "0.06076666666671713" 20:43:47 DEBUG "instance": "10.142.0.43:9100" 20:43:47 DEBUG } 20:43:47 DEBUG "0" 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG }, 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG "value": [ 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG ] 20:43:47 INFO [loop_until]: OK (rc = 0) 20:43:47 DEBUG --- stdout --- 20:43:47 DEBUG 1713642227, 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6284Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8606m 28% 7658Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17411m 58% 11331Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3372Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 10115m 33% 14314Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5191Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17237m 57% 11710Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19386m 64% 2973Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 20010m 67% 10597Mi 9% gke-xlou-cdm-ds-0f601751-j488 9204m 30% 7248Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 9260m 31% 15800Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 9181m 30% 7871Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 9808m 32% 13843Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11619m 73% 9987Mi 17% 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG "79706817.6" 20:43:47 DEBUG } 20:43:47 DEBUG --- stderr --- 20:43:47 DEBUG } 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG 20:43:47 DEBUG } 20:43:47 DEBUG ] 20:43:47 DEBUG } 20:43:47 DEBUG } 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642287" 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "26643.034584907058" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642287" 20:44:47 DEBUG --- status code --- 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "4766822400" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "instance": "10.142.0.107:9100" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "instance": "10.142.0.43:9100" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "0.04933333333333395" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 ERROR [http_cmd]: ERROR 20:44:47 ERROR --- status code --- 20:44:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:44:47 ERROR --- http response --- 20:44:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:44:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "ds-cts-0" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "ds-cts-2" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "0" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "3339.866666666667" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO 20:44:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:44:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "instance": "10.142.0.107:9100" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "instance": "10.142.0.43:9100" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "3771168533504" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "0" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "ds-cts-0" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "ds-cts-2" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "0" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "3339.9333333333334" 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642287" 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642287" 20:44:47 INFO 20:44:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642287" 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- status code --- 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- status code --- 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- status code --- 20:44:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG --- http response --- 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 INFO [http_cmd]: http status code OK 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG --- status code --- 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG { 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG { 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG --- http response --- 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG { 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG { 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:47 DEBUG { 20:44:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:47 DEBUG { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "data": { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "data": { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG ----- output ----- 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "data": { 20:44:47 DEBUG { 20:44:47 DEBUG "data": { 20:44:47 DEBUG { 20:44:47 DEBUG "data": { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "status": "success", 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG { 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "data": { 20:44:47 DEBUG "result": [ 20:44:47 INFO [loop_until]: OK (rc = 0) 20:44:47 DEBUG "data": { 20:44:47 DEBUG "result": [ 20:44:47 DEBUG { 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG { 20:44:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:47 DEBUG { 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG { 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG { 20:44:47 DEBUG --- stdout --- 20:44:47 DEBUG "resultType": "vector", 20:44:47 DEBUG { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG "metric": { 20:44:47 DEBUG }, 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "result": [ 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "result": [ 20:44:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17295m 8623Mi am-767f6b67b5-qcqz2 17495m 8819Mi am-767f6b67b5-vbltz 19515m 8394Mi ds-cts-0 10283m 11065Mi ds-cts-1 9567m 10778Mi ds-cts-2 10267m 11132Mi ds-idrepo-0 8211m 3901Mi ds-idrepo-1 8423m 3877Mi ds-idrepo-2 8342m 3838Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11410m 4451Mi 20:44:47 DEBUG "metric": { 20:44:47 DEBUG "pod": "ds-idrepo-0" 20:44:47 DEBUG "pod": "ds-cts-0" 20:44:47 DEBUG }, 20:44:47 DEBUG }, 20:44:47 DEBUG "instance": "10.142.0.107:9100" 20:44:47 DEBUG }, 20:44:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "pod": "ds-idrepo-0" 20:44:47 DEBUG "pod": "ds-idrepo-0" 20:44:47 DEBUG }, 20:44:47 DEBUG "instance": "10.142.0.107:9100" 20:44:47 DEBUG { 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG { 20:44:47 DEBUG --- stderr --- 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG }, 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "value": [ 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG }, 20:44:47 DEBUG 20:44:47 DEBUG }, 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG }, 20:44:47 DEBUG "metric": { 20:44:47 DEBUG }, 20:44:47 DEBUG "metric": { 20:44:47 DEBUG 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "instance": "10.142.0.107:9100" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "pod": "ds-idrepo-0" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG }, 20:44:47 DEBUG 20:44:47 DEBUG }, 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG [...] 20:44:47 DEBUG 20:44:47 DEBUG "pod": "ds-idrepo-2" 20:44:47 DEBUG }, 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:47 DEBUG 20:44:47 DEBUG }, 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "pod": "ds-cts-2" 20:44:47 DEBUG }, 20:44:47 DEBUG }, 20:44:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "pod": "ds-idrepo-2" 20:44:47 DEBUG "pod": "ds-cts-2" 20:44:47 DEBUG }, 20:44:47 DEBUG "instance": "10.142.0.43:9100" 20:44:47 DEBUG 20:44:47 DEBUG 20:44:47 DEBUG }, 20:44:47 DEBUG 20:44:47 DEBUG }, 20:44:47 DEBUG }, 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "instance": "10.142.0.43:9100" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "value": [ 20:44:47 DEBUG }, 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG }, 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG }, 20:44:47 DEBUG 20:44:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "0" 20:44:47 DEBUG }, 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "10.024218256550554" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "instance": "10.142.0.43:9100" 20:44:47 DEBUG }, 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "pod": "ds-idrepo-2" 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG ] 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "0.000033333333333333335" 20:44:47 DEBUG "0" 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG ] 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "11.346358891792251" 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG }, 20:44:47 DEBUG "value": [ 20:44:47 DEBUG "340174.5" 20:44:47 DEBUG }, 20:44:47 DEBUG "0.16" 20:44:47 DEBUG "519.8640894616011" 20:44:47 DEBUG } 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG "0" 20:44:47 DEBUG } 20:44:47 DEBUG "0" 20:44:47 DEBUG "0" 20:44:47 DEBUG ] 20:44:47 DEBUG "79584089.19999999" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG "3340.2" 20:44:47 DEBUG "value": [ 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG "79582280.5" 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG 1713642287, 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG "0.06563333333333787" 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG "0" 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG ] 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 DEBUG } 20:44:47 INFO [loop_until]: OK (rc = 0) 20:44:47 DEBUG --- stdout --- 20:44:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6275Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8577m 28% 7651Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 16930m 56% 11321Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 10321m 34% 14315Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17861m 59% 11690Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18896m 63% 2975Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 19987m 66% 10595Mi 9% gke-xlou-cdm-ds-0f601751-j488 9212m 30% 7237Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 9566m 32% 15814Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 9253m 30% 7842Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 9818m 32% 13848Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11611m 73% 9991Mi 17% 20:44:47 DEBUG --- stderr --- 20:44:47 DEBUG 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642347" 20:45:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "222762.92056831165" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "ds-cts-0" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "ds-cts-2" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "0" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 ERROR [http_cmd]: ERROR 20:45:47 DEBUG --- status code --- 20:45:47 ERROR --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 ERROR --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:45:47 DEBUG ----- output ----- 20:45:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:45:47 DEBUG { 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG "data": { 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:45:47 DEBUG }, 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:47 DEBUG "value": [ 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG 20:45:47 DEBUG { 20:45:47 DEBUG 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG 20:45:47 DEBUG "data": { 20:45:47 DEBUG [...] 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG 20:45:47 DEBUG "result": [ 20:45:47 DEBUG 20:45:47 DEBUG { 20:45:47 DEBUG 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:45:47 DEBUG "instance": "10.142.0.107:9100" 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 20:45:47 DEBUG "4773359616" 20:45:47 DEBUG 20:45:47 DEBUG ] 20:45:47 DEBUG 20:45:47 DEBUG } 20:45:47 DEBUG [...] 20:45:47 DEBUG ] 20:45:47 DEBUG 20:45:47 DEBUG } 20:45:47 DEBUG 20:45:47 DEBUG } 20:45:47 DEBUG 20:45:47 DEBUG "instance": "10.142.0.43:9100" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "0.037666666666666515" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "3332.9666666666667" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "ds-cts-0" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "ds-cts-2" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "0" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "3332.0333333333333" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "261.3" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "instance": "10.142.0.107:9100" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "instance": "10.142.0.43:9100" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "3774972476416" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "0" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "0" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:45:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "ds-idrepo-0" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "ds-cts-2" 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "0" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642347" 20:45:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642347" 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- status code --- 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG ----- output ----- 20:45:47 INFO [http_cmd]: http status code OK 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG { 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG --- status code --- 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "data": { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:47 DEBUG "data": { 20:45:47 DEBUG { 20:45:47 DEBUG "data": { 20:45:47 DEBUG "data": { 20:45:47 DEBUG "data": { 20:45:47 DEBUG "data": { 20:45:47 DEBUG "data": { 20:45:47 DEBUG --- http response --- 20:45:47 DEBUG "data": { 20:45:47 DEBUG "data": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "data": { 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:47 DEBUG "resultType": "vector", 20:45:47 INFO 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG { 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "data": { 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "result": [ 20:45:47 DEBUG ----- output ----- 20:45:47 DEBUG "result": [ 20:45:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 DEBUG { 20:45:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG { 20:45:47 DEBUG "data": { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "status": "success", 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "ds-idrepo-0" 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG "pod": "ds-cts-0" 20:45:47 DEBUG { 20:45:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:47 DEBUG "instance": "10.142.0.107:9100" 20:45:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:47 DEBUG "pod": "ds-idrepo-0" 20:45:47 DEBUG "data": { 20:45:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:47 DEBUG }, 20:45:47 DEBUG "pod": "ds-idrepo-0" 20:45:47 DEBUG "result": [ 20:45:47 DEBUG { 20:45:47 DEBUG "metric": { 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "resultType": "vector", 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG "metric": { 20:45:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG }, 20:45:47 DEBUG 20:45:47 DEBUG "result": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "instance": "10.142.0.107:9100" 20:45:47 DEBUG }, 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG { 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "metric": { 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG "instance": "10.142.0.107:9100" 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG [...] 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG }, 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 INFO [loop_until]: OK (rc = 0) 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "ds-idrepo-2" 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG --- stdout --- 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG }, 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:45:47 DEBUG 20:45:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17832m 8625Mi am-767f6b67b5-qcqz2 17986m 8823Mi am-767f6b67b5-vbltz 18865m 8979Mi ds-cts-0 12166m 11069Mi ds-cts-1 10993m 10779Mi ds-cts-2 10359m 11135Mi ds-idrepo-0 8362m 3882Mi ds-idrepo-1 8482m 3889Mi ds-idrepo-2 8368m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11319m 4457Mi 20:45:47 DEBUG [...] 20:45:47 DEBUG 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 20:45:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:45:47 DEBUG "instance": "10.142.0.43:9100" 20:45:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:45:47 DEBUG 20:45:47 DEBUG }, 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG }, 20:45:47 DEBUG --- stderr --- 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG "0" 20:45:47 DEBUG 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG "pod": "ds-idrepo-2" 20:45:47 DEBUG }, 20:45:47 DEBUG [...] 20:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:47 DEBUG "pod": "ds-idrepo-2" 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG 20:45:47 DEBUG ] 20:45:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG "pod": "ds-cts-2" 20:45:47 DEBUG "instance": "10.142.0.43:9100" 20:45:47 DEBUG } 20:45:47 DEBUG }, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "11.330128935499593" 20:45:47 DEBUG }, 20:45:47 DEBUG }, 20:45:47 DEBUG ] 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "22.582093366311685" 20:45:47 DEBUG "89897182.5" 20:45:47 DEBUG "0" 20:45:47 DEBUG "0.16" 20:45:47 DEBUG "0" 20:45:47 DEBUG 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG ] 20:45:47 DEBUG "value": [ 20:45:47 DEBUG "value": [ 20:45:47 DEBUG } 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG "instance": "10.142.0.43:9100" 20:45:47 DEBUG "536584.5" 20:45:47 DEBUG "0" 20:45:47 DEBUG } 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG } 20:45:47 DEBUG "3332.3333333333335" 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG }, 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG "0.0003919199999999999" 20:45:47 DEBUG "0.06709999999992627" 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG "value": [ 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG 1713642347, 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG "89899239.96666667" 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 DEBUG ] 20:45:47 DEBUG } 20:45:47 DEBUG } 20:45:47 INFO [loop_until]: OK (rc = 0) 20:45:47 DEBUG --- stdout --- 20:45:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6287Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8614m 28% 7633Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17640m 59% 11323Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3374Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 12255m 41% 14272Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5209Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 18319m 61% 11701Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19458m 65% 2970Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 20854m 69% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 9366m 31% 7232Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 11429m 38% 15771Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 9204m 30% 7872Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 11214m 37% 13824Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11569m 72% 9995Mi 17% 20:45:47 DEBUG --- stderr --- 20:45:47 DEBUG 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:46:47 DEBUG }, 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG "value": [ 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG 20:46:47 DEBUG { 20:46:47 DEBUG 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG 20:46:47 DEBUG "data": { 20:46:47 DEBUG [...] 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG 20:46:47 DEBUG "result": [ 20:46:47 DEBUG 20:46:47 DEBUG { 20:46:47 DEBUG 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:46:47 DEBUG "instance": "10.142.0.107:9100" 20:46:47 DEBUG }, 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG 20:46:47 DEBUG "4880338944" 20:46:47 DEBUG 20:46:47 DEBUG ] 20:46:47 DEBUG 20:46:47 DEBUG } 20:46:47 DEBUG [...] 20:46:47 DEBUG ] 20:46:47 DEBUG 20:46:47 DEBUG } 20:46:47 DEBUG 20:46:47 DEBUG } 20:46:47 DEBUG 20:46:47 DEBUG "instance": "10.142.0.43:9100" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0.04999999999999432" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 ERROR [http_cmd]: ERROR 20:46:47 ERROR --- status code --- 20:46:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:46:47 ERROR --- http response --- 20:46:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:46:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-cts-0" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "ds-cts-0" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-cts-2" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "3344.5" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "3344.1" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "ds-cts-0" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-cts-2" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "instance": "10.142.0.107:9100" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "instance": "10.142.0.43:9100" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "3779036920832" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "3338.2666666666664" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "ds-idrepo-0" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-cts-2" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "11.402972176230579" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO 20:46:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:46:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 DEBUG --- status code --- 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642407" 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG --- status code --- 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG { 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG --- http response --- 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG ----- output ----- 20:46:47 INFO [http_cmd]: http status code OK 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG "data": { 20:46:47 DEBUG { 20:46:47 DEBUG { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG { 20:46:47 DEBUG --- status code --- 20:46:47 DEBUG { 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG { 20:46:47 DEBUG "data": { 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG { 20:46:47 DEBUG "data": { 20:46:47 DEBUG --- http response --- 20:46:47 DEBUG "data": { 20:46:47 DEBUG { 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG { 20:46:47 DEBUG { 20:46:47 DEBUG "data": { 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "data": { 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "result": [ 20:46:47 DEBUG ----- output ----- 20:46:47 DEBUG "data": { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "data": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG { 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG { 20:46:47 DEBUG { 20:46:47 DEBUG { 20:46:47 DEBUG { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "ds-idrepo-0" 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "status": "success", 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "pod": "ds-idrepo-0" 20:46:47 DEBUG "result": [ 20:46:47 DEBUG { 20:46:47 DEBUG "instance": "10.142.0.107:9100" 20:46:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:47 DEBUG }, 20:46:47 DEBUG { 20:46:47 DEBUG "data": { 20:46:47 DEBUG "instance": "10.142.0.107:9100" 20:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:47 DEBUG "data": { 20:46:47 DEBUG { 20:46:47 DEBUG }, 20:46:47 DEBUG { 20:46:47 DEBUG "metric": { 20:46:47 DEBUG }, 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG }, 20:46:47 DEBUG }, 20:46:47 DEBUG "resultType": "vector", 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG "instance": "10.142.0.107:9100" 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "result": [ 20:46:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-idrepo-0" 20:46:47 DEBUG }, 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG }, 20:46:47 DEBUG { 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG { 20:46:47 DEBUG }, 20:46:47 DEBUG 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "metric": { 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "metric": { 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-cts-0" 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG }, 20:46:47 DEBUG [...] 20:46:47 DEBUG [...] 20:46:47 DEBUG }, 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-idrepo-2" 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-idrepo-2" 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:46:47 DEBUG }, 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG "instance": "10.142.0.43:9100" 20:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:47 DEBUG 20:46:47 DEBUG 20:46:47 DEBUG }, 20:46:47 DEBUG 20:46:47 DEBUG "instance": "10.142.0.43:9100" 20:46:47 DEBUG "pod": "ds-idrepo-2" 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG [...] 20:46:47 DEBUG }, 20:46:47 DEBUG }, 20:46:47 DEBUG [...] 20:46:47 DEBUG 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG }, 20:46:47 DEBUG }, 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "instance": "10.142.0.43:9100" 20:46:47 DEBUG 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG "pod": "ds-cts-0" 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG }, 20:46:47 DEBUG 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG 20:46:47 DEBUG }, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG }, 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 20:46:47 DEBUG "100482854.73333333" 20:46:47 DEBUG "0.16" 20:46:47 DEBUG 20:46:47 DEBUG "value": [ 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG } 20:46:47 DEBUG "value": [ 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:47 DEBUG ] 20:46:47 DEBUG ] 20:46:47 DEBUG "pod": "ds-cts-2" 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG "100480205.69999999" 20:46:47 DEBUG ] 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG "0.17749999999997687" 20:46:47 DEBUG }, 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG }, 20:46:47 DEBUG "0" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG "601317.5" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG "value": [ 20:46:47 DEBUG ] 20:46:47 DEBUG ] 20:46:47 DEBUG "value": [ 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG "3344.7" 20:46:47 DEBUG } 20:46:47 DEBUG 1713642407, 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG "0.39386666666666664" 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG ] 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 DEBUG } 20:46:47 INFO [loop_until]: OK (rc = 0) 20:46:47 DEBUG --- stdout --- 20:46:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21492m 8650Mi am-767f6b67b5-qcqz2 21819m 8851Mi am-767f6b67b5-vbltz 23940m 9041Mi ds-cts-0 19922m 11643Mi ds-cts-1 20555m 11555Mi ds-cts-2 20632m 11800Mi ds-idrepo-0 9381m 3882Mi ds-idrepo-1 9665m 3895Mi ds-idrepo-2 9769m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11179m 4563Mi 20:46:47 DEBUG --- stderr --- 20:46:47 DEBUG 20:46:47 INFO 20:46:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:46:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:46:47 INFO [loop_until]: OK (rc = 0) 20:46:47 DEBUG --- stdout --- 20:46:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10009m 33% 7634Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21751m 72% 11369Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20350m 68% 14869Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22076m 73% 11740Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19737m 66% 2979Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 23374m 78% 11260Mi 10% gke-xlou-cdm-ds-0f601751-j488 10408m 34% 7220Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20300m 67% 16518Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 10315m 34% 7861Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20594m 68% 14129Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11638m 73% 10093Mi 17% 20:46:47 DEBUG --- stderr --- 20:46:47 DEBUG 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642467" 20:47:47 ERROR [http_cmd]: ERROR 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 ERROR --- status code --- 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 ERROR --- http response --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:47:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:47:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG }, 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG "value": [ 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:47 DEBUG [...] 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG 20:47:47 DEBUG { 20:47:47 DEBUG 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG 20:47:47 DEBUG "data": { 20:47:47 DEBUG "pod": "ds-cts-0" 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG }, 20:47:47 DEBUG "result": [ 20:47:47 DEBUG "value": [ 20:47:47 DEBUG { 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "1618586.9656729554" 20:47:47 DEBUG "instance": "10.142.0.107:9100" 20:47:47 DEBUG ] 20:47:47 DEBUG }, 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642467" 20:47:47 DEBUG } 20:47:47 DEBUG "value": [ 20:47:47 DEBUG ] 20:47:47 DEBUG 20:47:47 DEBUG } 20:47:47 DEBUG 20:47:47 DEBUG } 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "instance": "10.142.0.43:9100" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0.05133333333333021" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "4951846912" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "ds-cts-0" 20:47:47 DEBUG "pod": "ds-cts-0" 20:47:47 DEBUG }, 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "ds-cts-2" 20:47:47 DEBUG "pod": "ds-cts-2" 20:47:47 DEBUG }, 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0" 20:47:47 DEBUG "0" 20:47:47 DEBUG ] 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "3346.6" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "3346.133333333333" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "instance": "10.142.0.107:9100" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "instance": "10.142.0.43:9100" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "3783183709184" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "3340.4333333333334" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "ds-cts-0" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "38283045.884198844" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "ds-idrepo-0" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "ds-cts-2" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "ds-idrepo-0" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "ds-idrepo-2" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "11.386592851789278" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO 20:47:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:47:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "instance": "10.142.0.107:9100" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "instance": "10.142.0.43:9100" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0.17699999999999536" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "3347.2999999999997" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "ds-cts-0" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "ds-cts-2" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0.4444" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "ds-cts-0" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "20.43894351536435" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642467" 20:47:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642467" 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG "data": { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "data": { 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "data": { 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG "result": [ 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "pod": "ds-idrepo-0" 20:47:47 DEBUG }, 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG "status": "success", 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG "pod": "ds-idrepo-0" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG "data": { 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG "result": [ 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG { 20:47:47 INFO [http_cmd]: http status code OK 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:47 INFO [loop_until]: OK (rc = 0) 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG "metric": { 20:47:47 DEBUG --- status code --- 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG --- stdout --- 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:47 DEBUG { 20:47:47 DEBUG { 20:47:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22062m 8663Mi am-767f6b67b5-qcqz2 22101m 8864Mi am-767f6b67b5-vbltz 24717m 9070Mi ds-cts-0 20320m 11887Mi ds-cts-1 20869m 11829Mi ds-cts-2 20899m 11866Mi ds-idrepo-0 9600m 3896Mi ds-idrepo-1 9743m 3911Mi ds-idrepo-2 9479m 3829Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11301m 4632Mi 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG }, 20:47:47 DEBUG --- http response --- 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG --- stderr --- 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "value": [ 20:47:47 DEBUG "data": { 20:47:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:47 DEBUG "data": { 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:47 DEBUG 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG ----- output ----- 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG 20:47:47 DEBUG "pod": "ds-idrepo-2" 20:47:47 DEBUG }, 20:47:47 DEBUG 20:47:47 DEBUG "result": [ 20:47:47 DEBUG { 20:47:47 DEBUG "result": [ 20:47:47 DEBUG "pod": "ds-idrepo-2" 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG { 20:47:47 DEBUG "status": "success", 20:47:47 DEBUG { 20:47:47 DEBUG }, 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG [...] 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "data": { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0.16" 20:47:47 DEBUG 20:47:47 DEBUG "instance": "10.142.0.107:9100" 20:47:47 DEBUG "resultType": "vector", 20:47:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "0" 20:47:47 DEBUG ] 20:47:47 DEBUG 20:47:47 DEBUG }, 20:47:47 DEBUG "result": [ 20:47:47 DEBUG }, 20:47:47 DEBUG "0" 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG 20:47:47 DEBUG "value": [ 20:47:47 DEBUG { 20:47:47 DEBUG "metric": { 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG "pod": "ds-cts-0" 20:47:47 DEBUG 20:47:47 DEBUG "instance": "10.142.0.107:9100" 20:47:47 DEBUG "value": [ 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG }, 20:47:47 DEBUG 20:47:47 DEBUG }, 20:47:47 DEBUG 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG } 20:47:47 DEBUG "65346177.33726226" 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG ] 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG } 20:47:47 DEBUG 20:47:47 DEBUG [...] 20:47:47 DEBUG 20:47:47 DEBUG ] 20:47:47 DEBUG "instance": "10.142.0.43:9100" 20:47:47 DEBUG 20:47:47 DEBUG 20:47:47 DEBUG } 20:47:47 DEBUG }, 20:47:47 DEBUG 20:47:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:47 DEBUG } 20:47:47 DEBUG "value": [ 20:47:47 DEBUG 20:47:47 DEBUG }, 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG "instance": "10.142.0.43:9100" 20:47:47 DEBUG "value": [ 20:47:47 DEBUG "100807873.46666667" 20:47:47 DEBUG }, 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG ] 20:47:47 DEBUG "value": [ 20:47:47 DEBUG "600941.5" 20:47:47 DEBUG } 20:47:47 DEBUG 1713642467, 20:47:47 DEBUG ] 20:47:47 DEBUG ] 20:47:47 DEBUG "100804153.63333333" 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG ] 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 DEBUG } 20:47:47 INFO 20:47:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:47:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:47:47 INFO [loop_until]: OK (rc = 0) 20:47:47 DEBUG --- stdout --- 20:47:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6287Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10108m 33% 7645Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21803m 73% 11380Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3372Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19727m 66% 15088Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 5217Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22152m 74% 11767Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19766m 66% 2987Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24698m 82% 11298Mi 10% gke-xlou-cdm-ds-0f601751-j488 10434m 34% 7234Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20306m 68% 16797Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10294m 34% 7869Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20719m 69% 14532Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11636m 73% 10161Mi 17% 20:47:47 DEBUG --- stderr --- 20:47:47 DEBUG 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642527" 20:48:47 ERROR [http_cmd]: ERROR 20:48:47 ERROR --- status code --- 20:48:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:48:47 ERROR --- http response --- 20:48:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:48:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "ds-cts-0" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "instance": "10.142.0.107:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "instance": "10.142.0.43:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0.05199999999998492" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "1497055.063217965" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "4958089216" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "ds-cts-0" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "3347.5" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "3345.6" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "ds-idrepo-0" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "instance": "10.142.0.107:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "instance": "10.142.0.43:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "3787367717888" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "32859530.39140295" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "3332.366666666667" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "ds-idrepo-0" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-idrepo-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "instance": "10.142.0.107:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "instance": "10.142.0.43:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0.1765000000000015" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "11.422236200422361" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "14.115651898115106" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "3344.7666666666664" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "ds-cts-0" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0.4678000000000002" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "ds-idrepo-0" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "ds-idrepo-2" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "0" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO 20:48:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:48:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642527" 20:48:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642527" 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "data": { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "result": [ 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "600864" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG ----- output ----- 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG { 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG "data": { 20:48:47 DEBUG { 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "status": "success", 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:47 DEBUG "result": [ 20:48:47 DEBUG "data": { 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG { 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "result": [ 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:47 DEBUG { 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG "data": { 20:48:47 DEBUG }, 20:48:47 DEBUG "metric": { 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "value": [ 20:48:47 DEBUG "instance": "10.142.0.107:9100" 20:48:47 DEBUG { 20:48:47 DEBUG "result": [ 20:48:47 DEBUG 20:48:47 DEBUG }, 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG { 20:48:47 DEBUG 20:48:47 DEBUG "value": [ 20:48:47 DEBUG "data": { 20:48:47 DEBUG "metric": { 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG "pod": "ds-idrepo-0" 20:48:47 DEBUG [...] 20:48:47 DEBUG 20:48:47 DEBUG "result": [ 20:48:47 DEBUG }, 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG { 20:48:47 DEBUG "value": [ 20:48:47 INFO [http_cmd]: http status code OK 20:48:47 DEBUG 20:48:47 DEBUG [...] 20:48:47 DEBUG "metric": { 20:48:47 DEBUG 20:48:47 DEBUG --- status code --- 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:48:47 DEBUG 20:48:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:47 DEBUG 20:48:47 DEBUG }, 20:48:47 DEBUG 20:48:47 DEBUG --- http response --- 20:48:47 DEBUG }, 20:48:47 DEBUG 20:48:47 DEBUG "value": [ 20:48:47 DEBUG [...] 20:48:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:47 DEBUG "value": [ 20:48:47 DEBUG "instance": "10.142.0.43:9100" 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG ----- output ----- 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG }, 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG { 20:48:47 DEBUG "0.16" 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 20:48:47 DEBUG 20:48:47 DEBUG "status": "success", 20:48:47 DEBUG ] 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG [...] 20:48:47 DEBUG "pod": "ds-idrepo-2" 20:48:47 DEBUG "data": { 20:48:47 DEBUG } 20:48:47 DEBUG "100981944.76666667" 20:48:47 DEBUG ] 20:48:47 DEBUG }, 20:48:47 DEBUG "resultType": "vector", 20:48:47 DEBUG ] 20:48:47 DEBUG 20:48:47 DEBUG } 20:48:47 DEBUG "value": [ 20:48:47 DEBUG "result": [ 20:48:47 DEBUG } 20:48:47 DEBUG 20:48:47 DEBUG ] 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG { 20:48:47 DEBUG } 20:48:47 DEBUG 20:48:47 DEBUG } 20:48:47 DEBUG "0" 20:48:47 DEBUG "metric": { 20:48:47 DEBUG "pod": "ds-cts-2" 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG "instance": "10.142.0.107:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG } 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG ] 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG } 20:48:47 DEBUG 20:48:47 DEBUG "47066541.52902987" 20:48:47 DEBUG } 20:48:47 DEBUG 20:48:47 DEBUG ] 20:48:47 DEBUG 20:48:47 DEBUG } 20:48:47 DEBUG [...] 20:48:47 DEBUG ] 20:48:47 DEBUG 20:48:47 DEBUG } 20:48:47 DEBUG 20:48:47 DEBUG } 20:48:47 DEBUG 20:48:47 DEBUG "instance": "10.142.0.43:9100" 20:48:47 DEBUG }, 20:48:47 DEBUG "value": [ 20:48:47 DEBUG 1713642527, 20:48:47 DEBUG "100984612.30000001" 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG ] 20:48:47 DEBUG } 20:48:47 DEBUG } 20:48:47 INFO [loop_until]: OK (rc = 0) 20:48:47 DEBUG --- stdout --- 20:48:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21904m 8666Mi am-767f6b67b5-qcqz2 22823m 8865Mi am-767f6b67b5-vbltz 24297m 9087Mi ds-cts-0 19772m 11909Mi ds-cts-1 21041m 12007Mi ds-cts-2 20961m 11979Mi ds-idrepo-0 9549m 3896Mi ds-idrepo-1 9519m 3915Mi ds-idrepo-2 9685m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11397m 4637Mi 20:48:47 DEBUG --- stderr --- 20:48:47 DEBUG 20:48:47 INFO 20:48:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:48:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:48:47 INFO [loop_until]: OK (rc = 0) 20:48:47 DEBUG --- stdout --- 20:48:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6286Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9968m 33% 7645Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21794m 72% 11399Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3372Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20173m 67% 15110Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5216Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22192m 74% 11761Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19787m 66% 2994Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24727m 82% 11330Mi 10% gke-xlou-cdm-ds-0f601751-j488 10487m 35% 7236Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20272m 67% 17012Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10268m 34% 7876Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20716m 69% 14654Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11652m 73% 10164Mi 17% 20:48:47 DEBUG --- stderr --- 20:48:47 DEBUG 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 ERROR [http_cmd]: ERROR 20:49:47 ERROR --- status code --- 20:49:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:49:47 ERROR --- http response --- 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:49:47 DEBUG --- status code --- 20:49:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642587" 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "lodemon-597977695c-ld9s4" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "1722321.5295095593" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "ds-cts-0" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "ds-cts-2" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "5040652288" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "instance": "10.142.0.107:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "instance": "10.142.0.43:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0.08100000000000972" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "ds-cts-0" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "ds-cts-2" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "3335.266666666667" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "lodemon-597977695c-ld9s4" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "3333.866666666667" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "instance": "10.142.0.107:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "instance": "10.142.0.43:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "3792228926464" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "ds-idrepo-0" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "ds-cts-2" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "3342.7" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "instance": "10.142.0.107:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "instance": "10.142.0.43:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0.3814333333333631" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "ds-idrepo-0" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "ds-idrepo-2" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "11.37675906756725" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "lodemon-597977695c-ld9s4" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0.0009886097711450505" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "3334.4" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "ds-cts-0" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "ds-cts-2" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0.5484999999999998" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "ds-idrepo-0" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "ds-idrepo-2" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "600883.5" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "instance": "10.142.0.107:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "instance": "10.142.0.43:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "100510250.96666667" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "ds-idrepo-0" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "ds-idrepo-2" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642587" 20:49:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642587" 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "instance": "10.142.0.107:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "instance": "10.142.0.43:9100" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "100507019.56666666" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0.16" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO [http_cmd]: http status code OK 20:49:47 DEBUG --- status code --- 20:49:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:47 DEBUG --- http response --- 20:49:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:47 DEBUG ----- output ----- 20:49:47 DEBUG { 20:49:47 DEBUG "status": "success", 20:49:47 DEBUG "data": { 20:49:47 DEBUG "resultType": "vector", 20:49:47 DEBUG "result": [ 20:49:47 DEBUG { 20:49:47 DEBUG "metric": { 20:49:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG [...] 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG 20:49:47 DEBUG "pod": "lodemon-597977695c-ld9s4" 20:49:47 DEBUG }, 20:49:47 DEBUG "value": [ 20:49:47 DEBUG 1713642587, 20:49:47 DEBUG "0" 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG ] 20:49:47 DEBUG } 20:49:47 DEBUG } 20:49:47 INFO 20:49:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:49:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:49:47 INFO [loop_until]: OK (rc = 0) 20:49:47 DEBUG --- stdout --- 20:49:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21860m 8666Mi am-767f6b67b5-qcqz2 22317m 8868Mi am-767f6b67b5-vbltz 24623m 9076Mi ds-cts-0 20466m 11908Mi ds-cts-1 20388m 12009Mi ds-cts-2 20501m 11981Mi ds-idrepo-0 9394m 3900Mi ds-idrepo-1 9559m 3914Mi ds-idrepo-2 9981m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11474m 4716Mi 20:49:47 DEBUG --- stderr --- 20:49:47 DEBUG 20:49:47 INFO 20:49:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:49:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:49:47 INFO [loop_until]: OK (rc = 0) 20:49:47 DEBUG --- stdout --- 20:49:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6283Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10023m 33% 7653Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21762m 72% 11392Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19914m 66% 15133Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 5220Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22162m 74% 11769Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19721m 66% 2999Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24676m 82% 11317Mi 10% gke-xlou-cdm-ds-0f601751-j488 10413m 34% 7226Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19793m 66% 17004Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10236m 34% 7891Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20371m 68% 14660Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11600m 73% 10238Mi 17% 20:49:47 DEBUG --- stderr --- 20:49:47 DEBUG 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 ERROR [http_cmd]: ERROR 20:50:47 ERROR --- status code --- 20:50:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:50:47 ERROR --- http response --- 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG --- status code --- 20:50:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "ds-cts-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-cts-2" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "ds-cts-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-cts-2" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG ----- output ----- 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG { 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG "data": { 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:50:47 DEBUG "data": { 20:50:47 DEBUG }, 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "value": [ 20:50:47 DEBUG "result": [ 20:50:47 DEBUG 20:50:47 DEBUG { 20:50:47 DEBUG 20:50:47 DEBUG "metric": { 20:50:47 DEBUG 20:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:47 DEBUG [...] 20:50:47 DEBUG }, 20:50:47 DEBUG 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:50:47 DEBUG 20:50:47 DEBUG }, 20:50:47 DEBUG [...] 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG 20:50:47 DEBUG "5046198272" 20:50:47 DEBUG 20:50:47 DEBUG ] 20:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:47 DEBUG } 20:50:47 DEBUG }, 20:50:47 DEBUG ] 20:50:47 DEBUG "value": [ 20:50:47 DEBUG } 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG } 20:50:47 DEBUG "3329.9333333333334" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "9290.569348575575" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "instance": "10.142.0.107:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "instance": "10.142.0.43:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0.07333333333333296" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-cts-2" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "3328.5" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "instance": "10.142.0.107:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "instance": "10.142.0.43:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "3797511171072" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG { 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "data": { 20:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG }, 20:50:47 DEBUG "result": [ 20:50:47 DEBUG "value": [ 20:50:47 DEBUG { 20:50:47 DEBUG 20:50:47 DEBUG "metric": { 20:50:47 DEBUG 20:50:47 DEBUG "instance": "10.142.0.107:9100" 20:50:47 DEBUG 20:50:47 DEBUG }, 20:50:47 DEBUG [...] 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:47 DEBUG [...] 20:50:47 DEBUG }, 20:50:47 DEBUG 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG 20:50:47 DEBUG "3353.5666666666666" 20:50:47 DEBUG "instance": "10.142.0.43:9100" 20:50:47 DEBUG ] 20:50:47 DEBUG }, 20:50:47 DEBUG } 20:50:47 DEBUG "value": [ 20:50:47 DEBUG ] 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG } 20:50:47 DEBUG "0.39653333333340773" 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-idrepo-2" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "11.354583690361364" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG { 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:50:47 DEBUG "metric": { 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:47 DEBUG { 20:50:47 DEBUG }, 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "value": [ 20:50:47 DEBUG "data": { 20:50:47 DEBUG 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG 20:50:47 DEBUG "result": [ 20:50:47 DEBUG 20:50:47 DEBUG { 20:50:47 DEBUG [...] 20:50:47 DEBUG "metric": { 20:50:47 DEBUG 20:50:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:47 DEBUG 20:50:47 DEBUG }, 20:50:47 DEBUG 20:50:47 DEBUG "value": [ 20:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:47 DEBUG 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG }, 20:50:47 DEBUG 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG [...] 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG "600592.5" 20:50:47 DEBUG 20:50:47 DEBUG ] 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:47 DEBUG 20:50:47 DEBUG } 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG 20:50:47 DEBUG ] 20:50:47 DEBUG { 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG } 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG }, 20:50:47 DEBUG } 20:50:47 DEBUG "data": { 20:50:47 DEBUG "value": [ 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "result": [ 20:50:47 DEBUG "5.334692803607465" 20:50:47 DEBUG { 20:50:47 DEBUG ] 20:50:47 DEBUG "metric": { 20:50:47 DEBUG } 20:50:47 DEBUG "pod": "ds-cts-0" 20:50:47 DEBUG ] 20:50:47 DEBUG }, 20:50:47 DEBUG } 20:50:47 DEBUG "value": [ 20:50:47 DEBUG } 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-cts-2" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0.5058000000000002" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "3327.4666666666667" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG ] 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG } 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG ] 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "instance": "10.142.0.107:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "instance": "10.142.0.43:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "100621701.36666667" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-idrepo-2" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-idrepo-2" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642647" 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "ds-idrepo-0" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "3389.8023298856833" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "0.16" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO [http_cmd]: http status code OK 20:50:47 DEBUG --- status code --- 20:50:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:47 DEBUG --- http response --- 20:50:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:47 DEBUG ----- output ----- 20:50:47 DEBUG { 20:50:47 DEBUG "status": "success", 20:50:47 DEBUG "data": { 20:50:47 DEBUG "resultType": "vector", 20:50:47 DEBUG "result": [ 20:50:47 DEBUG { 20:50:47 DEBUG "metric": { 20:50:47 DEBUG "instance": "10.142.0.107:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG [...] 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG 20:50:47 DEBUG "instance": "10.142.0.43:9100" 20:50:47 DEBUG }, 20:50:47 DEBUG "value": [ 20:50:47 DEBUG 1713642647, 20:50:47 DEBUG "100618890.33333334" 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG ] 20:50:47 DEBUG } 20:50:47 DEBUG } 20:50:47 INFO 20:50:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:50:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:50:47 INFO [loop_until]: OK (rc = 0) 20:50:47 DEBUG --- stdout --- 20:50:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22065m 8666Mi am-767f6b67b5-qcqz2 22415m 8866Mi am-767f6b67b5-vbltz 23815m 9073Mi ds-cts-0 19904m 11904Mi ds-cts-1 19398m 12015Mi ds-cts-2 20930m 12016Mi ds-idrepo-0 9534m 3900Mi ds-idrepo-1 9547m 3930Mi ds-idrepo-2 9619m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11523m 4718Mi 20:50:47 DEBUG --- stderr --- 20:50:47 DEBUG 20:50:47 INFO 20:50:47 INFO [loop_until]: kubectl --namespace=xlou top node 20:50:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:50:48 INFO [loop_until]: OK (rc = 0) 20:50:48 DEBUG --- stdout --- 20:50:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10008m 33% 7646Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21734m 72% 11384Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19875m 66% 15104Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 5210Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22111m 74% 11766Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19011m 63% 2995Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24517m 82% 11311Mi 10% gke-xlou-cdm-ds-0f601751-j488 10357m 34% 7243Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19829m 66% 17013Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10263m 34% 7875Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20534m 68% 14739Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11569m 72% 10243Mi 17% 20:50:48 DEBUG --- stderr --- 20:50:48 DEBUG 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 ERROR [http_cmd]: ERROR 20:51:47 DEBUG --- status code --- 20:51:47 ERROR --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 ERROR --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:51:47 DEBUG ----- output ----- 20:51:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "3344.1" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG "result": [ 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG { 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG "metric": { 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "11058.407501116239" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG ----- output ----- 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG { 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG "data": { 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:47 DEBUG "result": [ 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG }, 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "ds-cts-0" 20:51:47 DEBUG [...] 20:51:47 DEBUG "pod": "ds-cts-0" 20:51:47 DEBUG }, 20:51:47 DEBUG 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG }, 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG "value": [ 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "5051666432" 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG ] 20:51:47 DEBUG 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG } 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG }, 20:51:47 DEBUG ] 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG } 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG } 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "0" 20:51:47 DEBUG "0" 20:51:47 DEBUG ] 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "instance": "10.142.0.107:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "instance": "10.142.0.43:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "0.08166666666666438" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "3348.9333333333334" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "26737155.886292603" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "ds-idrepo-0" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "0" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "instance": "10.142.0.107:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "instance": "10.142.0.43:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "3802843558912" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "0" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:47 DEBUG "instance": "10.142.0.107:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG }, 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "value": [ 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG { 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG 20:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:47 DEBUG "data": { 20:51:47 DEBUG "instance": "10.142.0.43:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "result": [ 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG { 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "3355.3333333333335" 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "0.4165333333333185" 20:51:47 DEBUG ] 20:51:47 DEBUG "pod": "ds-idrepo-0" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG }, 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG "value": [ 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG 20:51:47 DEBUG } 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "ds-idrepo-2" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "0" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "11.414738134542116" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "instance": "10.142.0.107:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "instance": "10.142.0.43:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "100508046.3" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "600252" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG { 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG "data": { 20:51:47 DEBUG "data": { 20:51:47 DEBUG { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG "result": [ 20:51:47 DEBUG "data": { 20:51:47 DEBUG { 20:51:47 DEBUG { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "result": [ 20:51:47 DEBUG "pod": "ds-cts-0" 20:51:47 DEBUG "pod": "ds-idrepo-0" 20:51:47 DEBUG { 20:51:47 DEBUG }, 20:51:47 DEBUG }, 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG }, 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG "pod": "ds-idrepo-2" 20:51:47 DEBUG 20:51:47 DEBUG }, 20:51:47 DEBUG }, 20:51:47 DEBUG 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG }, 20:51:47 DEBUG "0.5231999999999999" 20:51:47 DEBUG "0" 20:51:47 DEBUG "value": [ 20:51:47 DEBUG ] 20:51:47 DEBUG ] 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG "7.713271292225665" 20:51:47 DEBUG ] 20:51:47 DEBUG ] 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "3348.7666666666664" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642707" 20:51:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642707" 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "ds-idrepo-0" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:47 DEBUG 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG 20:51:47 DEBUG { 20:51:47 DEBUG 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "pod": "ds-idrepo-2" 20:51:47 DEBUG "data": { 20:51:47 DEBUG }, 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "value": [ 20:51:47 DEBUG "result": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG { 20:51:47 DEBUG "0" 20:51:47 DEBUG "metric": { 20:51:47 DEBUG ] 20:51:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:47 DEBUG } 20:51:47 DEBUG }, 20:51:47 DEBUG ] 20:51:47 DEBUG "value": [ 20:51:47 DEBUG } 20:51:47 DEBUG 20:51:47 DEBUG } 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "0.16" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "pod": "ds-cts-2" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "32018236.618544426" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO [http_cmd]: http status code OK 20:51:47 DEBUG --- status code --- 20:51:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:47 DEBUG --- http response --- 20:51:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:47 DEBUG ----- output ----- 20:51:47 DEBUG { 20:51:47 DEBUG "status": "success", 20:51:47 DEBUG "data": { 20:51:47 DEBUG "resultType": "vector", 20:51:47 DEBUG "result": [ 20:51:47 DEBUG { 20:51:47 DEBUG "metric": { 20:51:47 DEBUG "instance": "10.142.0.107:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG [...] 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG 20:51:47 DEBUG "instance": "10.142.0.43:9100" 20:51:47 DEBUG }, 20:51:47 DEBUG "value": [ 20:51:47 DEBUG 1713642707, 20:51:47 DEBUG "100505628.9" 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG ] 20:51:47 DEBUG } 20:51:47 DEBUG } 20:51:47 INFO 20:51:47 INFO [loop_until]: kubectl --namespace=xlou top pods 20:51:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:51:48 INFO [loop_until]: OK (rc = 0) 20:51:48 DEBUG --- stdout --- 20:51:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21938m 8668Mi am-767f6b67b5-qcqz2 21703m 8869Mi am-767f6b67b5-vbltz 24338m 9089Mi ds-cts-0 20374m 11989Mi ds-cts-1 20134m 12009Mi ds-cts-2 20038m 12079Mi ds-idrepo-0 9455m 3900Mi ds-idrepo-1 9326m 3931Mi ds-idrepo-2 9693m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11246m 4722Mi 20:51:48 DEBUG --- stderr --- 20:51:48 DEBUG 20:51:48 INFO 20:51:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:51:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:51:48 INFO [loop_until]: OK (rc = 0) 20:51:48 DEBUG --- stdout --- 20:51:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10085m 33% 7644Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21125m 70% 11382Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3363Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19955m 66% 15206Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5213Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22287m 74% 11776Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19868m 66% 2981Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24826m 83% 11326Mi 10% gke-xlou-cdm-ds-0f601751-j488 10444m 34% 7237Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19610m 65% 17013Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10221m 34% 7902Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20417m 68% 14760Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11640m 73% 10244Mi 17% 20:51:48 DEBUG --- stderr --- 20:51:48 DEBUG 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "3351.0666666666666" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "ds-cts-0" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "ds-cts-2" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "ds-cts-0" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "ds-cts-2" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642767" 20:52:47 ERROR [http_cmd]: ERROR 20:52:47 ERROR --- status code --- 20:52:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:52:47 ERROR --- http response --- 20:52:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:52:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "358021739.39211553" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "5051068416" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "instance": "10.142.0.107:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "instance": "10.142.0.43:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0.08633333333333629" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG }, 20:52:47 DEBUG 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG 20:52:47 DEBUG }, 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG "value": [ 20:52:47 DEBUG }, 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG "3348.4" 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG ] 20:52:47 DEBUG "0" 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "ds-idrepo-0" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "ds-cts-2" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "instance": "10.142.0.107:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "instance": "10.142.0.43:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "3808044999680" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "ds-idrepo-0" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "ds-idrepo-2" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG { 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "3327.2" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "instance": "10.142.0.107:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "instance": "10.142.0.43:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0.4315999999999358" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "24.30550737285438" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "11.463800553137672" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "instance": "10.142.0.107:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "instance": "10.142.0.43:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "100305516.13333333" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "ds-cts-0" 20:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:47 DEBUG }, 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "ds-cts-2" 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0.5638333333333331" 20:52:47 DEBUG "600581.5" 20:52:47 DEBUG ] 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "instance": "10.142.0.107:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "instance": "10.142.0.43:9100" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "100304266.33333334" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "ds-idrepo-0" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "ds-idrepo-2" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0.16" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "3347.4666666666667" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "ds-idrepo-0" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "ds-idrepo-2" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "0" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642767" 20:52:47 INFO [http_cmd]: http status code OK 20:52:47 DEBUG --- status code --- 20:52:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:47 DEBUG --- http response --- 20:52:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:52:47 DEBUG ----- output ----- 20:52:47 DEBUG { 20:52:47 DEBUG "status": "success", 20:52:47 DEBUG "data": { 20:52:47 DEBUG "resultType": "vector", 20:52:47 DEBUG "result": [ 20:52:47 DEBUG { 20:52:47 DEBUG "metric": { 20:52:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG [...] 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG 20:52:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:47 DEBUG }, 20:52:47 DEBUG "value": [ 20:52:47 DEBUG 1713642767, 20:52:47 DEBUG "1232.6211254890159" 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG ] 20:52:47 DEBUG } 20:52:47 DEBUG } 20:52:48 INFO 20:52:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:52:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:52:48 INFO 20:52:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:52:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:52:48 INFO [loop_until]: OK (rc = 0) 20:52:48 DEBUG --- stdout --- 20:52:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21162m 8670Mi am-767f6b67b5-qcqz2 22514m 8867Mi am-767f6b67b5-vbltz 24904m 9076Mi ds-cts-0 20239m 12047Mi ds-cts-1 19520m 12013Mi ds-cts-2 20481m 12158Mi ds-idrepo-0 9365m 3900Mi ds-idrepo-1 9816m 3943Mi ds-idrepo-2 9761m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-597977695c-ld9s4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11422m 4723Mi 20:52:48 DEBUG --- stderr --- 20:52:48 DEBUG 20:52:48 INFO [loop_until]: OK (rc = 0) 20:52:48 DEBUG --- stdout --- 20:52:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9770m 32% 7652Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21878m 73% 11378Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19772m 66% 15267Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22254m 74% 11762Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19864m 66% 2986Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 23949m 80% 11321Mi 10% gke-xlou-cdm-ds-0f601751-j488 10422m 34% 7231Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19828m 66% 16993Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10259m 34% 7919Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20437m 68% 14849Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11687m 73% 10242Mi 17% 20:52:48 DEBUG --- stderr --- 20:52:48 DEBUG 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "3333.0666666666666" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642827" 20:53:47 ERROR [http_cmd]: ERROR 20:53:47 ERROR --- status code --- 20:53:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:53:47 ERROR --- http response --- 20:53:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:53:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "ds-cts-0" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-cts-2" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG "result": [ 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG { 20:53:47 DEBUG { 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "data": { 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG }, 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "value": [ 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "data": { 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-cts-0" 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG 20:53:47 DEBUG }, 20:53:47 DEBUG "result": [ 20:53:47 DEBUG [...] 20:53:47 DEBUG "value": [ 20:53:47 DEBUG { 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "metric": { 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "instance": "10.142.0.107:9100" 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG }, 20:53:47 DEBUG "pod": "ds-cts-0" 20:53:47 DEBUG [...] 20:53:47 DEBUG "value": [ 20:53:47 DEBUG }, 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "0" 20:53:47 DEBUG "pod": "ds-cts-2" 20:53:47 DEBUG [...] 20:53:47 DEBUG ] 20:53:47 DEBUG }, 20:53:47 DEBUG 20:53:47 DEBUG } 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG ] 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG 20:53:47 DEBUG } 20:53:47 DEBUG "0" 20:53:47 DEBUG "instance": "10.142.0.43:9100" 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG }, 20:53:47 DEBUG } 20:53:47 DEBUG "value": [ 20:53:47 DEBUG ] 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG } 20:53:47 DEBUG "0.06233333333332023" 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "5052616704" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG [...] 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "3331.0333333333333" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "ds-idrepo-0" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-cts-2" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "3332.133333333333" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "instance": "10.142.0.107:9100" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "instance": "10.142.0.43:9100" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "3812867298304" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG { 20:53:47 DEBUG { 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG "data": { 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG { 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "instance": "10.142.0.107:9100" 20:53:47 DEBUG "data": { 20:53:47 DEBUG "pod": "ds-idrepo-0" 20:53:47 DEBUG }, 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG "result": [ 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG { 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "metric": { 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "instance": "10.142.0.107:9100" 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG }, 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "instance": "10.142.0.43:9100" 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-idrepo-2" 20:53:47 DEBUG }, 20:53:47 DEBUG [...] 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "100393324.43333334" 20:53:47 DEBUG 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG "instance": "10.142.0.43:9100" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG }, 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG "value": [ 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 DEBUG "0.356133333333302" 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-cts-0" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "11.361105586644792" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "ds-cts-0" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-cts-2" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0.4527666666666661" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "600836.5" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "instance": "10.142.0.107:9100" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "instance": "10.142.0.43:9100" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "100390759" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "ds-idrepo-0" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-idrepo-2" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0.16" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "3331.3333333333335" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "ds-idrepo-0" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "ds-idrepo-2" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "0" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642827" 20:53:47 INFO [http_cmd]: http status code OK 20:53:47 DEBUG --- status code --- 20:53:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:47 DEBUG --- http response --- 20:53:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:47 DEBUG ----- output ----- 20:53:47 DEBUG { 20:53:47 DEBUG "status": "success", 20:53:47 DEBUG "data": { 20:53:47 DEBUG "resultType": "vector", 20:53:47 DEBUG "result": [ 20:53:47 DEBUG { 20:53:47 DEBUG "metric": { 20:53:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG [...] 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG 20:53:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:47 DEBUG }, 20:53:47 DEBUG "value": [ 20:53:47 DEBUG 1713642827, 20:53:47 DEBUG "515.2687360442809" 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG ] 20:53:47 DEBUG } 20:53:47 DEBUG } 20:53:48 INFO 20:53:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:53:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:53:48 INFO 20:53:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:53:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:53:48 INFO [loop_until]: OK (rc = 0) 20:53:48 DEBUG --- stdout --- 20:53:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21736m 8672Mi am-767f6b67b5-qcqz2 22240m 8867Mi am-767f6b67b5-vbltz 24023m 9084Mi ds-cts-0 19924m 12083Mi ds-cts-1 20208m 12014Mi ds-cts-2 20862m 12163Mi ds-idrepo-0 9442m 3900Mi ds-idrepo-1 9763m 3925Mi ds-idrepo-2 9615m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11514m 4726Mi 20:53:48 DEBUG --- stderr --- 20:53:48 DEBUG 20:53:48 INFO [loop_until]: OK (rc = 0) 20:53:48 DEBUG --- stdout --- 20:53:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9953m 33% 7659Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21770m 72% 11400Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20002m 66% 15310Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22159m 74% 11759Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19654m 65% 2980Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24570m 82% 11314Mi 10% gke-xlou-cdm-ds-0f601751-j488 10427m 34% 7234Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19779m 66% 17019Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10206m 34% 7909Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20714m 69% 14846Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11568m 72% 10243Mi 17% 20:53:48 DEBUG --- stderr --- 20:53:48 DEBUG 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642887" 20:54:47 ERROR [http_cmd]: ERROR 20:54:47 ERROR --- status code --- 20:54:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:54:47 ERROR --- http response --- 20:54:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:54:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "instance": "10.142.0.107:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "instance": "10.142.0.43:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0.08999999999998257" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:47 DEBUG "result": [ 20:54:47 DEBUG "result": [ 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG { 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "pod": "ds-cts-0" 20:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:47 DEBUG "data": { 20:54:47 DEBUG }, 20:54:47 DEBUG }, 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "value": [ 20:54:47 DEBUG "value": [ 20:54:47 DEBUG "result": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG { 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "metric": { 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:54:47 DEBUG [...] 20:54:47 DEBUG [...] 20:54:47 DEBUG }, 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-cts-2" 20:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:47 DEBUG 20:54:47 DEBUG }, 20:54:47 DEBUG }, 20:54:47 DEBUG [...] 20:54:47 DEBUG "value": [ 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG 20:54:47 DEBUG "0" 20:54:47 DEBUG "3325.7" 20:54:47 DEBUG 20:54:47 DEBUG ] 20:54:47 DEBUG ] 20:54:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 DEBUG }, 20:54:47 DEBUG ] 20:54:47 DEBUG ] 20:54:47 DEBUG "value": [ 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 DEBUG "5057392640" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG { 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG "data": { 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "ds-cts-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:54:47 DEBUG 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG 20:54:47 DEBUG { 20:54:47 DEBUG 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "pod": "ds-cts-2" 20:54:47 DEBUG "data": { 20:54:47 DEBUG }, 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "value": [ 20:54:47 DEBUG "result": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG { 20:54:47 DEBUG "0" 20:54:47 DEBUG "metric": { 20:54:47 DEBUG ] 20:54:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:47 DEBUG } 20:54:47 DEBUG }, 20:54:47 DEBUG ] 20:54:47 DEBUG "value": [ 20:54:47 DEBUG } 20:54:47 DEBUG 20:54:47 DEBUG } 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-idrepo-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "161648847.03378558" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-idrepo-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "3327.733333333333" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "ds-idrepo-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-cts-2" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "3355.1" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "instance": "10.142.0.107:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "instance": "10.142.0.43:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "3817983663104" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-idrepo-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "9.479611233042249" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "ds-idrepo-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-idrepo-2" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG ----- output ----- 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 DEBUG { 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "instance": "10.142.0.107:9100" 20:54:47 DEBUG "instance": "10.142.0.107:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "instance": "10.142.0.43:9100" 20:54:47 DEBUG "instance": "10.142.0.43:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "100210391.36666667" 20:54:47 DEBUG "0.4191333333333101" 20:54:47 DEBUG ] 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "600730.5" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "ds-cts-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-cts-2" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0.4975666666666674" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "11.330827157604705" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "instance": "10.142.0.107:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "instance": "10.142.0.43:9100" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "100207515.63333333" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "ds-idrepo-0" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "ds-idrepo-2" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "0.16" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642887" 20:54:47 INFO [http_cmd]: http status code OK 20:54:47 DEBUG --- status code --- 20:54:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:47 DEBUG --- http response --- 20:54:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:47 DEBUG ----- output ----- 20:54:47 DEBUG { 20:54:47 DEBUG "status": "success", 20:54:47 DEBUG "data": { 20:54:47 DEBUG "resultType": "vector", 20:54:47 DEBUG "result": [ 20:54:47 DEBUG { 20:54:47 DEBUG "metric": { 20:54:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG [...] 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG 20:54:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:47 DEBUG }, 20:54:47 DEBUG "value": [ 20:54:47 DEBUG 1713642887, 20:54:47 DEBUG "3332.4333333333334" 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG ] 20:54:47 DEBUG } 20:54:47 DEBUG } 20:54:48 INFO [http_cmd]: http status code OK 20:54:48 DEBUG --- status code --- 20:54:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:48 DEBUG --- http response --- 20:54:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:48 DEBUG ----- output ----- 20:54:48 DEBUG { 20:54:48 DEBUG "status": "success", 20:54:48 DEBUG "data": { 20:54:48 DEBUG "resultType": "vector", 20:54:48 DEBUG "result": [ 20:54:48 DEBUG { 20:54:48 DEBUG "metric": { 20:54:48 DEBUG "pod": "ds-idrepo-0" 20:54:48 DEBUG }, 20:54:48 DEBUG "value": [ 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG [...] 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG "pod": "ds-idrepo-2" 20:54:48 DEBUG }, 20:54:48 DEBUG "value": [ 20:54:48 DEBUG 1713642887, 20:54:48 DEBUG "0" 20:54:48 DEBUG ] 20:54:48 DEBUG } 20:54:48 DEBUG ] 20:54:48 DEBUG } 20:54:48 DEBUG } 20:54:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642888" 20:54:48 INFO [http_cmd]: http status code OK 20:54:48 DEBUG --- status code --- 20:54:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:48 DEBUG --- http response --- 20:54:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:54:48 DEBUG ----- output ----- 20:54:48 DEBUG { 20:54:48 DEBUG "status": "success", 20:54:48 DEBUG "data": { 20:54:48 DEBUG "resultType": "vector", 20:54:48 DEBUG "result": [ 20:54:48 DEBUG { 20:54:48 DEBUG "metric": { 20:54:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:48 DEBUG }, 20:54:48 DEBUG "value": [ 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG [...] 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG 20:54:48 DEBUG "pod": "ds-idrepo-0" 20:54:48 DEBUG }, 20:54:48 DEBUG "value": [ 20:54:48 DEBUG 1713642888, 20:54:48 DEBUG "4494.812450119713" 20:54:48 DEBUG ] 20:54:48 DEBUG } 20:54:48 DEBUG ] 20:54:48 DEBUG } 20:54:48 DEBUG } 20:54:48 INFO 20:54:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:54:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:54:48 INFO 20:54:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:54:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:54:48 INFO [loop_until]: OK (rc = 0) 20:54:48 DEBUG --- stdout --- 20:54:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22156m 8673Mi am-767f6b67b5-qcqz2 22068m 8873Mi am-767f6b67b5-vbltz 24259m 9088Mi ds-cts-0 20077m 12140Mi ds-cts-1 19596m 12045Mi ds-cts-2 20089m 12214Mi ds-idrepo-0 9472m 3900Mi ds-idrepo-1 9685m 3927Mi ds-idrepo-2 9703m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11348m 4729Mi 20:54:48 DEBUG --- stderr --- 20:54:48 DEBUG 20:54:48 INFO [loop_until]: OK (rc = 0) 20:54:48 DEBUG --- stdout --- 20:54:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6302Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9975m 33% 7644Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21807m 73% 11396Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19973m 66% 15358Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5187Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22134m 74% 11768Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19670m 65% 2976Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24584m 82% 11321Mi 10% gke-xlou-cdm-ds-0f601751-j488 10423m 34% 7216Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19420m 65% 17096Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10176m 34% 7904Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20596m 68% 14909Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11562m 72% 10246Mi 17% 20:54:48 DEBUG --- stderr --- 20:54:48 DEBUG 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG { 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG "data": { 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG "result": [ 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG { 20:55:47 DEBUG { 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:47 DEBUG "data": { 20:55:47 DEBUG }, 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "value": [ 20:55:47 DEBUG { 20:55:47 DEBUG "result": [ 20:55:47 DEBUG 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG { 20:55:47 DEBUG 20:55:47 DEBUG "data": { 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642947" 20:55:47 DEBUG "metric": { 20:55:47 DEBUG 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "pod": "ds-cts-0" 20:55:47 DEBUG [...] 20:55:47 DEBUG "result": [ 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG }, 20:55:47 DEBUG 20:55:47 DEBUG { 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG "metric": { 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG 20:55:47 DEBUG "pod": "lodemon-597977695c-ld9s4" 20:55:47 DEBUG }, 20:55:47 DEBUG 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 DEBUG }, 20:55:47 ERROR [http_cmd]: ERROR 20:55:47 DEBUG "value": [ 20:55:47 DEBUG [...] 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG "value": [ 20:55:47 ERROR --- status code --- 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG { 20:55:47 DEBUG 1713642947, 20:55:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "941.8885404101327" 20:55:47 ERROR --- http response --- 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "data": { 20:55:47 DEBUG ] 20:55:47 DEBUG [...] 20:55:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:55:47 DEBUG "pod": "ds-cts-2" 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:55:47 DEBUG }, 20:55:47 DEBUG "result": [ 20:55:47 DEBUG ] 20:55:47 DEBUG 20:55:47 DEBUG "value": [ 20:55:47 DEBUG { 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG 1713642947, 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG "metric": { 20:55:47 DEBUG } 20:55:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:47 DEBUG "0" 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:55:47 DEBUG }, 20:55:47 DEBUG ] 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG } 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG ] 20:55:47 DEBUG 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 DEBUG "3342.5333333333333" 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG { 20:55:47 DEBUG } 20:55:47 DEBUG [...] 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG ] 20:55:47 DEBUG 20:55:47 DEBUG "data": { 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG "result": [ 20:55:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:55:47 DEBUG { 20:55:47 DEBUG }, 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "value": [ 20:55:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG }, 20:55:47 DEBUG "5059485696" 20:55:47 DEBUG "value": [ 20:55:47 DEBUG ] 20:55:47 DEBUG 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG ] 20:55:47 DEBUG 20:55:47 DEBUG } 20:55:47 DEBUG [...] 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "lodemon-597977695c-ld9s4" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "0" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "instance": "10.142.0.107:9100" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "instance": "10.142.0.43:9100" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "0.0773333333333331" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "3341" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "ds-cts-0" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "ds-cts-2" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "2.5" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "ds-idrepo-0" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "ds-cts-2" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "0" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "0" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "3346.8333333333335" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "instance": "10.142.0.107:9100" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "instance": "10.142.0.43:9100" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "3823269288960" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "ds-idrepo-0" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "ds-idrepo-2" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "0" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "22.049899885775435" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "11.477160635031145" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642947" 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:47 DEBUG "instance": "10.142.0.107:9100" 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG }, 20:55:47 DEBUG { 20:55:47 DEBUG "value": [ 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG 20:55:47 DEBUG "data": { 20:55:47 DEBUG 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG 20:55:47 DEBUG "result": [ 20:55:47 DEBUG [...] 20:55:47 DEBUG { 20:55:47 DEBUG 20:55:47 DEBUG "metric": { 20:55:47 DEBUG 20:55:47 DEBUG "instance": "10.142.0.107:9100" 20:55:47 DEBUG 20:55:47 DEBUG }, 20:55:47 DEBUG "instance": "10.142.0.43:9100" 20:55:47 DEBUG "value": [ 20:55:47 DEBUG }, 20:55:47 DEBUG 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG 20:55:47 DEBUG "0.40666666666661605" 20:55:47 DEBUG [...] 20:55:47 DEBUG ] 20:55:47 DEBUG 20:55:47 DEBUG } 20:55:47 DEBUG 20:55:47 DEBUG ] 20:55:47 DEBUG 20:55:47 DEBUG } 20:55:47 DEBUG "instance": "10.142.0.43:9100" 20:55:47 DEBUG } 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "100470068.7" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "599984.5" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642947" 20:55:47 INFO [http_cmd]: http status code OK 20:55:47 DEBUG --- status code --- 20:55:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:47 DEBUG --- http response --- 20:55:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:47 DEBUG ----- output ----- 20:55:47 DEBUG { 20:55:47 DEBUG "status": "success", 20:55:47 DEBUG "data": { 20:55:47 DEBUG "resultType": "vector", 20:55:47 DEBUG "result": [ 20:55:47 DEBUG { 20:55:47 DEBUG "metric": { 20:55:47 DEBUG "pod": "ds-cts-0" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG [...] 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG 20:55:47 DEBUG "pod": "ds-cts-2" 20:55:47 DEBUG }, 20:55:47 DEBUG "value": [ 20:55:47 DEBUG 1713642947, 20:55:47 DEBUG "0.5629333333333335" 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG ] 20:55:47 DEBUG } 20:55:47 DEBUG } 20:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642948" 20:55:48 INFO [http_cmd]: http status code OK 20:55:48 DEBUG --- status code --- 20:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:48 DEBUG --- http response --- 20:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:48 DEBUG ----- output ----- 20:55:48 DEBUG { 20:55:48 DEBUG "status": "success", 20:55:48 DEBUG "data": { 20:55:48 DEBUG "resultType": "vector", 20:55:48 DEBUG "result": [ 20:55:48 DEBUG { 20:55:48 DEBUG "metric": { 20:55:48 DEBUG "pod": "ds-idrepo-0" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG [...] 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG "pod": "ds-idrepo-2" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 1713642948, 20:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713642948" 20:55:48 DEBUG "0" 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG } 20:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642948" 20:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713642948" 20:55:48 INFO [http_cmd]: http status code OK 20:55:48 DEBUG --- status code --- 20:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:48 DEBUG --- http response --- 20:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:48 DEBUG ----- output ----- 20:55:48 DEBUG { 20:55:48 DEBUG "status": "success", 20:55:48 DEBUG "data": { 20:55:48 DEBUG "resultType": "vector", 20:55:48 DEBUG "result": [ 20:55:48 DEBUG { 20:55:48 DEBUG "metric": { 20:55:48 DEBUG "instance": "10.142.0.107:9100" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG [...] 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG "instance": "10.142.0.43:9100" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 1713642948, 20:55:48 DEBUG "100467405.13333333" 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG } 20:55:48 INFO [http_cmd]: http status code OK 20:55:48 DEBUG --- status code --- 20:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:48 DEBUG --- http response --- 20:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:48 DEBUG ----- output ----- 20:55:48 DEBUG { 20:55:48 DEBUG "status": "success", 20:55:48 DEBUG "data": { 20:55:48 DEBUG "resultType": "vector", 20:55:48 DEBUG "result": [ 20:55:48 DEBUG { 20:55:48 DEBUG "metric": { 20:55:48 DEBUG "pod": "ds-idrepo-0" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG [...] 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG "pod": "ds-idrepo-2" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 1713642948, 20:55:48 DEBUG "0" 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG } 20:55:48 INFO [http_cmd]: http status code OK 20:55:48 DEBUG --- status code --- 20:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:48 DEBUG --- http response --- 20:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:48 DEBUG ----- output ----- 20:55:48 DEBUG { 20:55:48 DEBUG "status": "success", 20:55:48 DEBUG "data": { 20:55:48 DEBUG "resultType": "vector", 20:55:48 DEBUG "result": [ 20:55:48 DEBUG { 20:55:48 DEBUG "metric": { 20:55:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG [...] 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 1713642948, 20:55:48 DEBUG "0.16" 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG } 20:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642948" 20:55:48 INFO [http_cmd]: http status code OK 20:55:48 DEBUG --- status code --- 20:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:48 DEBUG --- http response --- 20:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:48 DEBUG ----- output ----- 20:55:48 DEBUG { 20:55:48 DEBUG "status": "success", 20:55:48 DEBUG "data": { 20:55:48 DEBUG "resultType": "vector", 20:55:48 DEBUG "result": [ 20:55:48 DEBUG { 20:55:48 DEBUG "metric": { 20:55:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG [...] 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 1713642948, 20:55:48 DEBUG "3341.0333333333333" 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG } 20:55:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713642948" 20:55:48 INFO [http_cmd]: http status code OK 20:55:48 DEBUG --- status code --- 20:55:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:48 DEBUG --- http response --- 20:55:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:55:48 DEBUG ----- output ----- 20:55:48 DEBUG { 20:55:48 DEBUG "status": "success", 20:55:48 DEBUG "data": { 20:55:48 DEBUG "resultType": "vector", 20:55:48 DEBUG "result": [ 20:55:48 DEBUG { 20:55:48 DEBUG "metric": { 20:55:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG [...] 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG 20:55:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:48 DEBUG }, 20:55:48 DEBUG "value": [ 20:55:48 DEBUG 1713642948, 20:55:48 DEBUG "3302.5599677484383" 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG ] 20:55:48 DEBUG } 20:55:48 DEBUG } 20:55:48 INFO 20:55:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:55:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:55:48 INFO 20:55:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:55:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:55:48 INFO [loop_until]: OK (rc = 0) 20:55:48 DEBUG --- stdout --- 20:55:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21686m 8675Mi am-767f6b67b5-qcqz2 21905m 8877Mi am-767f6b67b5-vbltz 24964m 9080Mi ds-cts-0 19924m 12222Mi ds-cts-1 19647m 12093Mi ds-cts-2 20031m 12221Mi ds-idrepo-0 9664m 3900Mi ds-idrepo-1 9447m 3927Mi ds-idrepo-2 9756m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11290m 4731Mi 20:55:48 DEBUG --- stderr --- 20:55:48 DEBUG 20:55:48 INFO [loop_until]: OK (rc = 0) 20:55:48 DEBUG --- stdout --- 20:55:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6301Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10036m 33% 7652Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21873m 73% 11400Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20187m 67% 15429Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22324m 74% 11775Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19856m 66% 2976Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24755m 82% 11313Mi 10% gke-xlou-cdm-ds-0f601751-j488 10477m 35% 7241Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19686m 65% 17098Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10224m 34% 7910Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20329m 68% 14912Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11646m 73% 10245Mi 17% 20:55:48 DEBUG --- stderr --- 20:55:48 DEBUG 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 ERROR [http_cmd]: ERROR 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643007" 20:56:47 ERROR --- status code --- 20:56:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:56:47 ERROR --- http response --- 20:56:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:56:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "ds-idrepo-0" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "0" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "instance": "10.142.0.107:9100" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "instance": "10.142.0.43:9100" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "0.09066666666668273" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643007" 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "ds-cts-2" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "0" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "3333" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "ds-cts-0" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "ds-cts-2" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "0" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "5062197248" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "0" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "3335.0333333333333" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "ds-cts-0" 20:56:47 DEBUG "pod": "ds-idrepo-0" 20:56:47 DEBUG }, 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "ds-cts-2" 20:56:47 DEBUG "pod": "ds-cts-2" 20:56:47 DEBUG }, 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "30.933333333333334" 20:56:47 DEBUG "0" 20:56:47 DEBUG ] 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643007" 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "3339.133333333333" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643007" 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "instance": "10.142.0.107:9100" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "instance": "10.142.0.43:9100" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "3828368823296" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643007" 20:56:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643007" 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "pod": "ds-idrepo-0" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "pod": "ds-idrepo-2" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "0" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG --- http response --- 20:56:47 INFO [http_cmd]: http status code OK 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:47 DEBUG --- status code --- 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:47 DEBUG { 20:56:47 DEBUG --- http response --- 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "11.350024186197292" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:47 DEBUG ----- output ----- 20:56:47 DEBUG { 20:56:47 DEBUG "status": "success", 20:56:47 DEBUG "data": { 20:56:47 DEBUG "resultType": "vector", 20:56:47 DEBUG "result": [ 20:56:47 DEBUG { 20:56:47 DEBUG "metric": { 20:56:47 DEBUG "instance": "10.142.0.107:9100" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG [...] 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG 20:56:47 DEBUG "instance": "10.142.0.43:9100" 20:56:47 DEBUG }, 20:56:47 DEBUG "value": [ 20:56:47 DEBUG 1713643007, 20:56:47 DEBUG "0.4270333333333089" 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG ] 20:56:47 DEBUG } 20:56:47 DEBUG } 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "pod": "ds-cts-2" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643007, 20:56:48 DEBUG "0" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643008" 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643008" 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "599853" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "instance": "10.142.0.107:9100" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "instance": "10.142.0.43:9100" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "100118448.13333333" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643008" 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643008" 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "ds-cts-0" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "pod": "ds-cts-2" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "0.5765333333333331" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "ds-idrepo-0" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "pod": "ds-idrepo-2" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "0" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643008" 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "ds-idrepo-0" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "pod": "ds-idrepo-2" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "0" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643008" 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643008" 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG { 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG "data": { 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:48 DEBUG }, 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG "value": [ 20:56:48 DEBUG { 20:56:48 DEBUG 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG 20:56:48 DEBUG "data": { 20:56:48 DEBUG 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG [...] 20:56:48 DEBUG "result": [ 20:56:48 DEBUG 20:56:48 DEBUG { 20:56:48 DEBUG 20:56:48 DEBUG "metric": { 20:56:48 DEBUG 20:56:48 DEBUG "instance": "10.142.0.107:9100" 20:56:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:48 DEBUG }, 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG 20:56:48 DEBUG "0.16" 20:56:48 DEBUG 20:56:48 DEBUG ] 20:56:48 DEBUG [...] 20:56:48 DEBUG } 20:56:48 DEBUG 20:56:48 DEBUG ] 20:56:48 DEBUG 20:56:48 DEBUG } 20:56:48 DEBUG 20:56:48 DEBUG } 20:56:48 DEBUG "instance": "10.142.0.43:9100" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "100114525.30000001" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643008" 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "3335.733333333333" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643008" 20:56:48 INFO [http_cmd]: http status code OK 20:56:48 DEBUG --- status code --- 20:56:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:48 DEBUG --- http response --- 20:56:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:56:48 DEBUG ----- output ----- 20:56:48 DEBUG { 20:56:48 DEBUG "status": "success", 20:56:48 DEBUG "data": { 20:56:48 DEBUG "resultType": "vector", 20:56:48 DEBUG "result": [ 20:56:48 DEBUG { 20:56:48 DEBUG "metric": { 20:56:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG [...] 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG 20:56:48 DEBUG "pod": "ds-idrepo-0" 20:56:48 DEBUG }, 20:56:48 DEBUG "value": [ 20:56:48 DEBUG 1713643008, 20:56:48 DEBUG "3090.5834296724465" 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG ] 20:56:48 DEBUG } 20:56:48 DEBUG } 20:56:48 INFO 20:56:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:56:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:56:48 INFO 20:56:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:56:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:56:48 INFO [loop_until]: OK (rc = 0) 20:56:48 DEBUG --- stdout --- 20:56:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6302Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10037m 33% 7671Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21725m 72% 11403Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3368Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19973m 66% 15434Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22273m 74% 11770Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19670m 65% 2989Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24621m 82% 11315Mi 10% gke-xlou-cdm-ds-0f601751-j488 10399m 34% 7243Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19625m 65% 17106Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10208m 34% 7899Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20380m 68% 14987Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11578m 72% 10245Mi 17% 20:56:48 DEBUG --- stderr --- 20:56:48 DEBUG 20:56:48 INFO [loop_until]: OK (rc = 0) 20:56:48 DEBUG --- stdout --- 20:56:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21950m 8674Mi am-767f6b67b5-qcqz2 21674m 8872Mi am-767f6b67b5-vbltz 24229m 9076Mi ds-cts-0 19911m 12218Mi ds-cts-1 19436m 12098Mi ds-cts-2 20186m 12294Mi ds-idrepo-0 9376m 3925Mi ds-idrepo-1 9745m 3936Mi ds-idrepo-2 9579m 3830Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11268m 4733Mi 20:56:48 DEBUG --- stderr --- 20:56:48 DEBUG 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643067" 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643067" 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "pod": "ds-cts-0" 20:57:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:47 DEBUG }, 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "pod": "ds-cts-2" 20:57:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:47 DEBUG }, 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "0" 20:57:47 DEBUG "3332.1666666666665" 20:57:47 DEBUG ] 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 ERROR [http_cmd]: ERROR 20:57:47 ERROR --- status code --- 20:57:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:57:47 ERROR --- http response --- 20:57:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:57:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "5064974336" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "0" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643067" 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "instance": "10.142.0.107:9100" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "instance": "10.142.0.43:9100" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "0.08099999999999075" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "198678.3762550582" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "3332.7" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "pod": "ds-cts-0" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "pod": "ds-cts-2" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "28.933333333333334" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "pod": "ds-idrepo-0" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "pod": "ds-cts-2" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "0" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643067" 20:57:47 INFO [http_cmd]: http status code OK 20:57:47 DEBUG --- status code --- 20:57:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:47 DEBUG --- http response --- 20:57:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:47 DEBUG ----- output ----- 20:57:47 DEBUG { 20:57:47 DEBUG "status": "success", 20:57:47 DEBUG "data": { 20:57:47 DEBUG "resultType": "vector", 20:57:47 DEBUG "result": [ 20:57:47 DEBUG { 20:57:47 DEBUG "metric": { 20:57:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG [...] 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG 20:57:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:47 DEBUG }, 20:57:47 DEBUG "value": [ 20:57:47 DEBUG 1713643067, 20:57:47 DEBUG "0" 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG ] 20:57:47 DEBUG } 20:57:47 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "3334.133333333333" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643068" 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "instance": "10.142.0.107:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "instance": "10.142.0.43:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "3833648315392" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643068" 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643068" 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "ds-idrepo-0" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "ds-idrepo-2" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "0" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "11.41942135190947" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG { 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "instance": "10.142.0.107:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "instance": "10.142.0.43:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "0.41389999999998867" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "22.738874505256334" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "600312" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "instance": "10.142.0.107:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "instance": "10.142.0.43:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "100392378.66666666" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "ds-cts-0" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "ds-cts-2" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "0.49546666666666583" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "ds-idrepo-0" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "ds-idrepo-2" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "0" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "instance": "10.142.0.107:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "instance": "10.142.0.43:9100" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "100389916.5" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "ds-idrepo-0" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "ds-idrepo-2" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "0" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "0.16" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "3333" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643068" 20:57:48 INFO [http_cmd]: http status code OK 20:57:48 DEBUG --- status code --- 20:57:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:48 DEBUG --- http response --- 20:57:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:48 DEBUG ----- output ----- 20:57:48 DEBUG { 20:57:48 DEBUG "status": "success", 20:57:48 DEBUG "data": { 20:57:48 DEBUG "resultType": "vector", 20:57:48 DEBUG "result": [ 20:57:48 DEBUG { 20:57:48 DEBUG "metric": { 20:57:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG [...] 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG 20:57:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 20:57:48 DEBUG }, 20:57:48 DEBUG "value": [ 20:57:48 DEBUG 1713643068, 20:57:48 DEBUG "0" 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG ] 20:57:48 DEBUG } 20:57:48 DEBUG } 20:57:48 INFO 20:57:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:57:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:57:48 INFO 20:57:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:57:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:57:48 INFO [loop_until]: OK (rc = 0) 20:57:48 DEBUG --- stdout --- 20:57:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 6292Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10001m 33% 7693Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21728m 72% 11396Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20049m 67% 15444Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5197Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22226m 74% 11772Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19773m 66% 2986Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24754m 82% 11333Mi 10% gke-xlou-cdm-ds-0f601751-j488 10093m 33% 7269Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19616m 65% 17197Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10230m 34% 7911Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20530m 68% 14990Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11648m 73% 10247Mi 17% 20:57:48 DEBUG --- stderr --- 20:57:48 DEBUG 20:57:48 INFO [loop_until]: OK (rc = 0) 20:57:48 DEBUG --- stdout --- 20:57:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21747m 8677Mi am-767f6b67b5-qcqz2 22163m 8878Mi am-767f6b67b5-vbltz 23907m 9086Mi ds-cts-0 19965m 12226Mi ds-cts-1 20570m 12195Mi ds-cts-2 20157m 12343Mi ds-idrepo-0 9413m 3942Mi ds-idrepo-1 9354m 3935Mi ds-idrepo-2 9687m 3870Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11060m 4737Mi 20:57:48 DEBUG --- stderr --- 20:57:48 DEBUG 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643127" 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 ERROR [http_cmd]: ERROR 20:58:47 ERROR --- status code --- 20:58:47 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:58:47 ERROR --- http response --- 20:58:47 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:58:47 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "3350.2" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "pod": "ds-idrepo-0" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "pod": "ds-cts-2" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "0" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "3329.633333333333" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "pod": "ds-cts-0" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "pod": "ds-cts-2" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "28.3" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "pod": "ds-cts-0" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "69962317.51628977" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643127" 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "pod": "ds-cts-0" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "pod": "ds-cts-2" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "0" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "5063180288" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643127" 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "instance": "10.142.0.107:9100" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "instance": "10.142.0.43:9100" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "0.08133333333334274" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: http status code OK 20:58:47 DEBUG --- status code --- 20:58:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:47 DEBUG --- http response --- 20:58:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:58:47 DEBUG ----- output ----- 20:58:47 DEBUG { 20:58:47 DEBUG "status": "success", 20:58:47 DEBUG "data": { 20:58:47 DEBUG "resultType": "vector", 20:58:47 DEBUG "result": [ 20:58:47 DEBUG { 20:58:47 DEBUG "metric": { 20:58:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG [...] 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG 20:58:47 DEBUG "pod": "ds-cts-0" 20:58:47 DEBUG }, 20:58:47 DEBUG "value": [ 20:58:47 DEBUG 1713643127, 20:58:47 DEBUG "491126256.496742" 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG ] 20:58:47 DEBUG } 20:58:47 DEBUG } 20:58:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643127" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643127, 20:58:48 DEBUG "0" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643128" 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "3349.7999999999997" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643128" 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:48 DEBUG --- status code --- 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG { 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:58:48 DEBUG "data": { 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "data": { 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "instance": "10.142.0.107:9100" 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG }, 20:58:48 DEBUG "result": [ 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "value": [ 20:58:48 DEBUG { 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:48 DEBUG 20:58:48 DEBUG "instance": "10.142.0.107:9100" 20:58:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:58:48 DEBUG }, 20:58:48 DEBUG 20:58:48 DEBUG }, 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG [...] 20:58:48 DEBUG "value": [ 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG [...] 20:58:48 DEBUG "instance": "10.142.0.43:9100" 20:58:48 DEBUG [...] 20:58:48 DEBUG [...] 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG 20:58:48 DEBUG }, 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG "pod": "ds-cts-0" 20:58:48 DEBUG "0.40083333333344906" 20:58:48 DEBUG "instance": "10.142.0.43:9100" 20:58:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:58:48 DEBUG { 20:58:48 DEBUG }, 20:58:48 DEBUG ] 20:58:48 DEBUG }, 20:58:48 DEBUG }, 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "value": [ 20:58:48 DEBUG } 20:58:48 DEBUG "value": [ 20:58:48 DEBUG "value": [ 20:58:48 DEBUG "data": { 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG ] 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "19.791680720581034" 20:58:48 DEBUG } 20:58:48 DEBUG "3838859383808" 20:58:48 DEBUG "11.361271558671415" 20:58:48 DEBUG "result": [ 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG ] 20:58:48 DEBUG { 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 DEBUG "metric": { 20:58:48 DEBUG ] 20:58:48 DEBUG ] 20:58:48 DEBUG ] 20:58:48 DEBUG "pod": "ds-idrepo-0" 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 DEBUG }, 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "ds-idrepo-2" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "0" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "instance": "10.142.0.107:9100" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "instance": "10.142.0.43:9100" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "100310925.8" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643128" 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "600026" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "instance": "10.142.0.107:9100" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "instance": "10.142.0.43:9100" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "100314536.43333334" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "ds-cts-0" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "ds-cts-2" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "0.5639000000000001" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "ds-idrepo-0" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "ds-idrepo-2" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "0" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "0.16" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "ds-idrepo-0" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "ds-idrepo-2" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "0" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "3349.9" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643128" 20:58:48 INFO [http_cmd]: http status code OK 20:58:48 DEBUG --- status code --- 20:58:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:48 DEBUG --- http response --- 20:58:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:48 DEBUG ----- output ----- 20:58:48 DEBUG { 20:58:48 DEBUG "status": "success", 20:58:48 DEBUG "data": { 20:58:48 DEBUG "resultType": "vector", 20:58:48 DEBUG "result": [ 20:58:48 DEBUG { 20:58:48 DEBUG "metric": { 20:58:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG [...] 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG 20:58:48 DEBUG "pod": "ds-cts-0" 20:58:48 DEBUG }, 20:58:48 DEBUG "value": [ 20:58:48 DEBUG 1713643128, 20:58:48 DEBUG "83693572.7098505" 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG ] 20:58:48 DEBUG } 20:58:48 DEBUG } 20:58:48 INFO 20:58:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:58:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:58:48 INFO 20:58:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:58:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:58:48 INFO [loop_until]: OK (rc = 0) 20:58:48 DEBUG --- stdout --- 20:58:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6303Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9951m 33% 7691Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21776m 72% 11405Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19840m 66% 15437Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5215Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22131m 74% 11781Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19637m 65% 2979Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 23929m 80% 11321Mi 10% gke-xlou-cdm-ds-0f601751-j488 10352m 34% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19508m 65% 17205Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10165m 34% 7903Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20328m 68% 15032Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11577m 72% 10244Mi 17% 20:58:48 DEBUG --- stderr --- 20:58:48 DEBUG 20:58:48 INFO [loop_until]: OK (rc = 0) 20:58:48 DEBUG --- stdout --- 20:58:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22049m 8677Mi am-767f6b67b5-qcqz2 22410m 8876Mi am-767f6b67b5-vbltz 23727m 9085Mi ds-cts-0 19696m 12216Mi ds-cts-1 19696m 12198Mi ds-cts-2 20222m 12334Mi ds-idrepo-0 9439m 3942Mi ds-idrepo-1 9348m 3935Mi ds-idrepo-2 9513m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11291m 4737Mi 20:58:48 DEBUG --- stderr --- 20:58:48 DEBUG 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643187" 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643187" 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643187" 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643187" 20:59:47 INFO [http_cmd]: http status code OK 20:59:47 DEBUG --- status code --- 20:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:47 DEBUG --- http response --- 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643187" 20:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:59:47 DEBUG ----- output ----- 20:59:47 DEBUG { 20:59:47 DEBUG "status": "success", 20:59:47 DEBUG "data": { 20:59:47 DEBUG "resultType": "vector", 20:59:47 DEBUG "result": [ 20:59:47 DEBUG { 20:59:47 DEBUG "metric": { 20:59:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:47 DEBUG }, 20:59:47 DEBUG "value": [ 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG [...] 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG "pod": "ds-idrepo-0" 20:59:47 DEBUG }, 20:59:47 DEBUG "value": [ 20:59:47 DEBUG 1713643187, 20:59:47 DEBUG "0" 20:59:47 DEBUG ] 20:59:47 DEBUG } 20:59:47 DEBUG ] 20:59:47 DEBUG } 20:59:47 DEBUG } 20:59:47 INFO [http_cmd]: http status code OK 20:59:47 DEBUG --- status code --- 20:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:47 DEBUG --- http response --- 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643187" 20:59:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:47 DEBUG ----- output ----- 20:59:47 DEBUG { 20:59:47 DEBUG "status": "success", 20:59:47 DEBUG "data": { 20:59:47 DEBUG "resultType": "vector", 20:59:47 DEBUG "result": [ 20:59:47 DEBUG { 20:59:47 DEBUG "metric": { 20:59:47 DEBUG "pod": "ds-cts-0" 20:59:47 DEBUG }, 20:59:47 DEBUG "value": [ 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG [...] 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG 20:59:47 DEBUG "pod": "ds-cts-2" 20:59:47 DEBUG }, 20:59:47 DEBUG "value": [ 20:59:47 DEBUG 1713643187, 20:59:47 DEBUG "29.7" 20:59:47 DEBUG ] 20:59:47 DEBUG } 20:59:47 DEBUG ] 20:59:47 DEBUG } 20:59:47 DEBUG } 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643187" 20:59:47 INFO [http_cmd]: http status code OK 20:59:47 DEBUG --- status code --- 20:59:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:47 DEBUG --- http response --- 20:59:47 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:59:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643187" 20:59:48 ERROR [http_cmd]: ERROR 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 ERROR --- status code --- 20:59:48 DEBUG --- status code --- 20:59:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 ERROR --- http response --- 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:59:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643187, 20:59:48 DEBUG "3343.1" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:59:48 DEBUG }, 20:59:48 DEBUG "pod": "ds-idrepo-0" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:48 DEBUG 20:59:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:59:48 DEBUG }, 20:59:48 DEBUG "pod": "ds-cts-2" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643187, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643187, 20:59:48 DEBUG "3340.2666666666664" 20:59:48 DEBUG 1713643187, 20:59:48 DEBUG "5067169792" 20:59:48 DEBUG ] 20:59:48 DEBUG "0" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG { 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "ds-cts-0" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "ds-cts-2" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "instance": "10.142.0.107:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "instance": "10.142.0.43:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0.07466666666667453" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "ds-cts-0" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "9081.081081081082" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "instance": "10.142.0.107:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "instance": "10.142.0.43:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0.3839333333332869" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "3325.0333333333333" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "instance": "10.142.0.107:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "instance": "10.142.0.43:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "100127094.19999999" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "instance": "10.142.0.107:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "instance": "10.142.0.43:9100" 20:59:48 DEBUG }, 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG "value": [ 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG "3843999562752" 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "11.376364063351009" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "ds-idrepo-0" 20:59:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:48 DEBUG }, 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "ds-idrepo-2" 20:59:48 DEBUG "pod": "ds-cts-0" 20:59:48 DEBUG }, 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0" 20:59:48 DEBUG "20.21693591638502" 20:59:48 DEBUG ] 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "600332.5" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "ds-idrepo-0" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "ds-idrepo-2" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "instance": "10.142.0.107:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "instance": "10.142.0.43:9100" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "100122317.66666666" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "ds-idrepo-0" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "ds-idrepo-2" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "ds-cts-0" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "ds-cts-2" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0.5300333333333327" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "0.16" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "3340.0666666666666" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643188" 20:59:48 INFO [http_cmd]: http status code OK 20:59:48 DEBUG --- status code --- 20:59:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:48 DEBUG --- http response --- 20:59:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:48 DEBUG ----- output ----- 20:59:48 DEBUG { 20:59:48 DEBUG "status": "success", 20:59:48 DEBUG "data": { 20:59:48 DEBUG "resultType": "vector", 20:59:48 DEBUG "result": [ 20:59:48 DEBUG { 20:59:48 DEBUG "metric": { 20:59:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG [...] 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG 20:59:48 DEBUG "pod": "ds-cts-0" 20:59:48 DEBUG }, 20:59:48 DEBUG "value": [ 20:59:48 DEBUG 1713643188, 20:59:48 DEBUG "80078486.4864865" 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG ] 20:59:48 DEBUG } 20:59:48 DEBUG } 20:59:48 INFO 20:59:48 INFO [loop_until]: kubectl --namespace=xlou top node 20:59:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:59:48 INFO 20:59:48 INFO [loop_until]: kubectl --namespace=xlou top pods 20:59:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:59:49 INFO [loop_until]: OK (rc = 0) 20:59:49 DEBUG --- stdout --- 20:59:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21993m 8678Mi am-767f6b67b5-qcqz2 21684m 8877Mi am-767f6b67b5-vbltz 24315m 9086Mi ds-cts-0 20542m 12216Mi ds-cts-1 19868m 12201Mi ds-cts-2 20261m 12337Mi ds-idrepo-0 9472m 3941Mi ds-idrepo-1 9458m 3928Mi ds-idrepo-2 9600m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11272m 4741Mi 20:59:49 DEBUG --- stderr --- 20:59:49 DEBUG 20:59:49 INFO [loop_until]: OK (rc = 0) 20:59:49 DEBUG --- stdout --- 20:59:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9699m 32% 7699Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21887m 73% 11410Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20229m 67% 15429Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5184Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22209m 74% 11769Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19681m 65% 2987Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24760m 82% 11308Mi 10% gke-xlou-cdm-ds-0f601751-j488 10414m 34% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19894m 66% 17209Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10259m 34% 7895Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19806m 66% 15024Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11575m 72% 10245Mi 17% 20:59:49 DEBUG --- stderr --- 20:59:49 DEBUG 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 ERROR [http_cmd]: ERROR 21:00:48 ERROR --- status code --- 21:00:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:00:48 ERROR --- http response --- 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG --- status code --- 21:00:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:00:48 DEBUG "status": "success", 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG "data": { 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG "data": { 21:00:48 DEBUG { 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "metric": { 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG "result": [ 21:00:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG { 21:00:48 DEBUG { 21:00:48 DEBUG }, 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "value": [ 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG "data": { 21:00:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:00:48 DEBUG 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG }, 21:00:48 DEBUG 21:00:48 DEBUG "result": [ 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG { 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG "metric": { 21:00:48 DEBUG { 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-cts-0" 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG }, 21:00:48 DEBUG "data": { 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG "value": [ 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG 21:00:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:48 DEBUG 21:00:48 DEBUG "result": [ 21:00:48 DEBUG 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:48 DEBUG }, 21:00:48 DEBUG 21:00:48 DEBUG { 21:00:48 DEBUG 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG { 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG [...] 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "3319.866666666667" 21:00:48 DEBUG 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG "data": { 21:00:48 DEBUG ] 21:00:48 DEBUG 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG } 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "0" 21:00:48 DEBUG "result": [ 21:00:48 DEBUG ] 21:00:48 DEBUG "pod": "ds-cts-2" 21:00:48 DEBUG 21:00:48 DEBUG ] 21:00:48 DEBUG { 21:00:48 DEBUG } 21:00:48 DEBUG }, 21:00:48 DEBUG 21:00:48 DEBUG } 21:00:48 DEBUG "metric": { 21:00:48 DEBUG } 21:00:48 DEBUG "value": [ 21:00:48 DEBUG [...] 21:00:48 DEBUG ] 21:00:48 DEBUG "instance": "10.142.0.107:9100" 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG 21:00:48 DEBUG } 21:00:48 DEBUG }, 21:00:48 DEBUG "30.866666666666667" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 DEBUG } 21:00:48 DEBUG "value": [ 21:00:48 DEBUG ] 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG } 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG "pod": "ds-cts-2" 21:00:48 DEBUG 21:00:48 DEBUG ] 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG }, 21:00:48 DEBUG 21:00:48 DEBUG } 21:00:48 DEBUG "value": [ 21:00:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:00:48 DEBUG [...] 21:00:48 DEBUG } 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG 21:00:48 DEBUG "0" 21:00:48 DEBUG 21:00:48 DEBUG ] 21:00:48 DEBUG 21:00:48 DEBUG } 21:00:48 DEBUG "instance": "10.142.0.43:9100" 21:00:48 DEBUG ] 21:00:48 DEBUG }, 21:00:48 DEBUG } 21:00:48 DEBUG "value": [ 21:00:48 DEBUG } 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0.08099999999999832" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "ds-cts-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-cts-2" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "3316.7" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "5069119488" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "19053.910785763223" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "instance": "10.142.0.107:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "instance": "10.142.0.43:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "3849195738112" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "instance": "10.142.0.107:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "instance": "10.142.0.43:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0.4123333333332899" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-idrepo-2" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "3354.5" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "instance": "10.142.0.107:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "instance": "10.142.0.43:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "100223009.63333333" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "11.321458597036695" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "instance": "10.142.0.107:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "instance": "10.142.0.43:9100" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "100226659.5" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "9.430435974180902" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "600427.5" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-idrepo-2" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-idrepo-2" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "ds-cts-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-cts-2" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0.5263666666666665" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "0.16" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "3321.9333333333334" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643248" 21:00:48 INFO [http_cmd]: http status code OK 21:00:48 DEBUG --- status code --- 21:00:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:48 DEBUG --- http response --- 21:00:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:00:48 DEBUG ----- output ----- 21:00:48 DEBUG { 21:00:48 DEBUG "status": "success", 21:00:48 DEBUG "data": { 21:00:48 DEBUG "resultType": "vector", 21:00:48 DEBUG "result": [ 21:00:48 DEBUG { 21:00:48 DEBUG "metric": { 21:00:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG [...] 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG 21:00:48 DEBUG "pod": "ds-idrepo-0" 21:00:48 DEBUG }, 21:00:48 DEBUG "value": [ 21:00:48 DEBUG 1713643248, 21:00:48 DEBUG "4283.437308427391" 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG ] 21:00:48 DEBUG } 21:00:48 DEBUG } 21:00:49 INFO 21:00:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:00:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:00:49 INFO 21:00:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:00:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:00:49 INFO [loop_until]: OK (rc = 0) 21:00:49 DEBUG --- stdout --- 21:00:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22477m 8675Mi am-767f6b67b5-qcqz2 21896m 8878Mi am-767f6b67b5-vbltz 24202m 9085Mi ds-cts-0 20072m 12215Mi ds-cts-1 19346m 12197Mi ds-cts-2 20512m 12335Mi ds-idrepo-0 9447m 3942Mi ds-idrepo-1 9544m 3927Mi ds-idrepo-2 9746m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11128m 4743Mi 21:00:49 DEBUG --- stderr --- 21:00:49 DEBUG 21:00:49 INFO [loop_until]: OK (rc = 0) 21:00:49 DEBUG --- stdout --- 21:00:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6280Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9934m 33% 7700Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21714m 72% 11405Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19987m 66% 15442Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5150Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22133m 74% 11774Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19622m 65% 2981Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24579m 82% 11321Mi 10% gke-xlou-cdm-ds-0f601751-j488 10351m 34% 7281Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19473m 65% 17209Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10195m 34% 7897Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20434m 68% 15032Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11533m 72% 10248Mi 17% 21:00:49 DEBUG --- stderr --- 21:00:49 DEBUG 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "ds-cts-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-cts-2" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "5070274560" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "ds-cts-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-cts-2" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "29.333333333333332" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "instance": "10.142.0.107:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "instance": "10.142.0.43:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0.09266666666667332" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 ERROR [http_cmd]: ERROR 21:01:48 ERROR --- status code --- 21:01:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:01:48 ERROR --- http response --- 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:01:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-cts-2" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "3324.0333333333333" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "3336.9333333333334" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "128154058.55029017" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-idrepo-2" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "3345.8333333333335" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "instance": "10.142.0.107:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "instance": "10.142.0.43:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "3854574990336" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "instance": "10.142.0.107:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "instance": "10.142.0.43:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0.42473333333335955" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "instance": "10.142.0.107:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "instance": "10.142.0.43:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "100333344.76666667" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "11.330564861186271" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643308" 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "instance": "10.142.0.107:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "instance": "10.142.0.43:9100" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "100337774.93333334" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:48 DEBUG }, 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:48 DEBUG }, 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "5.96315071696218" 21:01:48 DEBUG "599896" 21:01:48 DEBUG ] 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-idrepo-2" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-idrepo-2" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0.16" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "ds-cts-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-cts-2" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "0.5245666666666664" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "3324.2" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643308" 21:01:48 INFO [http_cmd]: http status code OK 21:01:48 DEBUG --- status code --- 21:01:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:48 DEBUG --- http response --- 21:01:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:48 DEBUG ----- output ----- 21:01:48 DEBUG { 21:01:48 DEBUG "status": "success", 21:01:48 DEBUG "data": { 21:01:48 DEBUG "resultType": "vector", 21:01:48 DEBUG "result": [ 21:01:48 DEBUG { 21:01:48 DEBUG "metric": { 21:01:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG [...] 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG 21:01:48 DEBUG "pod": "ds-idrepo-0" 21:01:48 DEBUG }, 21:01:48 DEBUG "value": [ 21:01:48 DEBUG 1713643308, 21:01:48 DEBUG "2675.6690432764053" 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG ] 21:01:48 DEBUG } 21:01:48 DEBUG } 21:01:49 INFO 21:01:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:01:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:01:49 INFO [loop_until]: OK (rc = 0) 21:01:49 DEBUG --- stdout --- 21:01:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22127m 8680Mi am-767f6b67b5-qcqz2 21757m 8875Mi am-767f6b67b5-vbltz 24636m 9083Mi ds-cts-0 20231m 12219Mi ds-cts-1 19393m 12203Mi ds-cts-2 20652m 12338Mi ds-idrepo-0 9425m 3942Mi ds-idrepo-1 9632m 3928Mi ds-idrepo-2 9669m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11319m 4744Mi 21:01:49 DEBUG --- stderr --- 21:01:49 DEBUG 21:01:49 INFO 21:01:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:01:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:01:49 INFO [loop_until]: OK (rc = 0) 21:01:49 DEBUG --- stdout --- 21:01:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6286Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9977m 33% 7711Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21791m 72% 11399Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3351Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19979m 66% 15439Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 5172Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22151m 74% 11771Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19712m 66% 2994Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24556m 82% 11313Mi 10% gke-xlou-cdm-ds-0f601751-j488 10329m 34% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19611m 65% 17200Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10163m 34% 7894Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20470m 68% 15037Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11577m 72% 10245Mi 17% 21:01:49 DEBUG --- stderr --- 21:01:49 DEBUG 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "instance": "10.142.0.107:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "instance": "10.142.0.43:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0.07100000000000173" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 ERROR [http_cmd]: ERROR 21:02:48 ERROR --- status code --- 21:02:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:02:48 ERROR --- http response --- 21:02:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:02:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "ds-cts-0" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "ds-cts-2" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "ds-cts-0" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "ds-cts-2" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "31.266666666666666" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "5073612800" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "3339.366666666667" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "3319.733333333333" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "ds-idrepo-0" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "ds-cts-2" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "149028510.7635381" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "ds-idrepo-0" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "ds-idrepo-2" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "3332.633333333333" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "instance": "10.142.0.107:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "instance": "10.142.0.43:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "100121088.5" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "instance": "10.142.0.107:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "instance": "10.142.0.43:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "3859871263744" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "instance": "10.142.0.107:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "instance": "10.142.0.43:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0.38093333333339574" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0.16" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "11.440959140071696" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "instance": "10.142.0.107:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "instance": "10.142.0.43:9100" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "100122899.26666667" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:02:48 DEBUG }, 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:02:48 DEBUG }, 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "599536" 21:02:48 DEBUG "22.78693568649806" 21:02:48 DEBUG ] 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "ds-idrepo-0" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "ds-idrepo-2" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "ds-idrepo-0" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "ds-idrepo-2" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "ds-cts-0" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "ds-cts-2" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0.5328666666666664" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "3318.7" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643368" 21:02:48 INFO [http_cmd]: http status code OK 21:02:48 DEBUG --- status code --- 21:02:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:48 DEBUG --- http response --- 21:02:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:02:48 DEBUG ----- output ----- 21:02:48 DEBUG { 21:02:48 DEBUG "status": "success", 21:02:48 DEBUG "data": { 21:02:48 DEBUG "resultType": "vector", 21:02:48 DEBUG "result": [ 21:02:48 DEBUG { 21:02:48 DEBUG "metric": { 21:02:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG [...] 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG 21:02:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:02:48 DEBUG }, 21:02:48 DEBUG "value": [ 21:02:48 DEBUG 1713643368, 21:02:48 DEBUG "0" 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG ] 21:02:48 DEBUG } 21:02:48 DEBUG } 21:02:49 INFO 21:02:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:02:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:02:49 INFO [loop_until]: OK (rc = 0) 21:02:49 DEBUG --- stdout --- 21:02:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21621m 8678Mi am-767f6b67b5-qcqz2 21824m 8877Mi am-767f6b67b5-vbltz 24669m 9107Mi ds-cts-0 20180m 12219Mi ds-cts-1 19295m 12205Mi ds-cts-2 19929m 12334Mi ds-idrepo-0 9451m 3942Mi ds-idrepo-1 9538m 3927Mi ds-idrepo-2 9854m 3870Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 6m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11177m 4747Mi 21:02:49 DEBUG --- stderr --- 21:02:49 DEBUG 21:02:49 INFO 21:02:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:02:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:02:49 INFO [loop_until]: OK (rc = 0) 21:02:49 DEBUG --- stdout --- 21:02:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6285Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9944m 33% 7693Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21796m 73% 11398Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19367m 64% 15445Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 21454m 71% 11776Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19725m 66% 2980Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24650m 82% 11340Mi 10% gke-xlou-cdm-ds-0f601751-j488 10353m 34% 7280Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19634m 65% 17209Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10211m 34% 7897Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20393m 68% 15027Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11681m 73% 10246Mi 17% 21:02:49 DEBUG --- stderr --- 21:02:49 DEBUG 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "instance": "10.142.0.107:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "instance": "10.142.0.43:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0.07966666666666053" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "ds-idrepo-0" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "ds-cts-2" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 ERROR [http_cmd]: ERROR 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 ERROR --- status code --- 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 ERROR --- http response --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:03:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG { 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "data": { 21:03:48 DEBUG "data": { 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG "resultType": "vector", 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG "result": [ 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG "result": [ 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG { 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG { 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "metric": { 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "data": { 21:03:48 DEBUG "pod": "ds-cts-0" 21:03:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:03:48 DEBUG "pod": "ds-cts-0" 21:03:48 DEBUG "resultType": "vector", 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG }, 21:03:48 DEBUG }, 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG }, 21:03:48 DEBUG "result": [ 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG "value": [ 21:03:48 DEBUG "value": [ 21:03:48 DEBUG { 21:03:48 DEBUG "value": [ 21:03:48 DEBUG { 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG 21:03:48 DEBUG "metric": { 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "data": { 21:03:48 DEBUG 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG 21:03:48 DEBUG }, 21:03:48 DEBUG [...] 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG [...] 21:03:48 DEBUG "result": [ 21:03:48 DEBUG [...] 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG { 21:03:48 DEBUG 21:03:48 DEBUG { 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG 21:03:48 DEBUG "metric": { 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "data": { 21:03:48 DEBUG 21:03:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "ds-cts-2" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "pod": "ds-cts-2" 21:03:48 DEBUG }, 21:03:48 DEBUG "pod": "ds-cts-2" 21:03:48 DEBUG [...] 21:03:48 DEBUG }, 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG "result": [ 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG }, 21:03:48 DEBUG 21:03:48 DEBUG "value": [ 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG { 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG "metric": { 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG 21:03:48 DEBUG "29.966666666666665" 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG "0" 21:03:48 DEBUG 21:03:48 DEBUG "0" 21:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:48 DEBUG ] 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG }, 21:03:48 DEBUG ] 21:03:48 DEBUG [...] 21:03:48 DEBUG ] 21:03:48 DEBUG }, 21:03:48 DEBUG } 21:03:48 DEBUG { 21:03:48 DEBUG "value": [ 21:03:48 DEBUG } 21:03:48 DEBUG 21:03:48 DEBUG } 21:03:48 DEBUG "value": [ 21:03:48 DEBUG ] 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG 21:03:48 DEBUG ] 21:03:48 DEBUG 21:03:48 DEBUG ] 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG } 21:03:48 DEBUG "data": { 21:03:48 DEBUG 21:03:48 DEBUG } 21:03:48 DEBUG 21:03:48 DEBUG } 21:03:48 DEBUG "3337.1666666666665" 21:03:48 DEBUG } 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG 21:03:48 DEBUG } 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG "result": [ 21:03:48 DEBUG [...] 21:03:48 DEBUG }, 21:03:48 DEBUG } 21:03:48 DEBUG { 21:03:48 DEBUG 21:03:48 DEBUG "value": [ 21:03:48 DEBUG ] 21:03:48 DEBUG "metric": { 21:03:48 DEBUG 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG } 21:03:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:03:48 DEBUG 21:03:48 DEBUG "0" 21:03:48 DEBUG } 21:03:48 DEBUG }, 21:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:48 DEBUG ] 21:03:48 DEBUG "value": [ 21:03:48 DEBUG }, 21:03:48 DEBUG } 21:03:48 DEBUG 21:03:48 DEBUG "value": [ 21:03:48 DEBUG ] 21:03:48 DEBUG 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG } 21:03:48 DEBUG 21:03:48 DEBUG "3331.5666666666666" 21:03:48 DEBUG } 21:03:48 DEBUG [...] 21:03:48 DEBUG ] 21:03:48 DEBUG 21:03:48 DEBUG } 21:03:48 DEBUG 21:03:48 DEBUG ] 21:03:48 DEBUG 21:03:48 DEBUG } 21:03:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:03:48 DEBUG } 21:03:48 DEBUG }, 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "5079719936" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "ds-idrepo-0" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "ds-idrepo-2" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "3336.8333333333335" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "instance": "10.142.0.107:9100" 21:03:48 DEBUG "instance": "10.142.0.107:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "instance": "10.142.0.43:9100" 21:03:48 DEBUG "instance": "10.142.0.43:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "3864956324864" 21:03:48 DEBUG "100054350.23333332" 21:03:48 DEBUG ] 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "instance": "10.142.0.107:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "instance": "10.142.0.43:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0.39973333333335953" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0.16" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "11.382585149675402" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "instance": "10.142.0.107:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "instance": "10.142.0.43:9100" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "100058089.56666666" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "599510" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "ds-cts-2" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "ds-idrepo-0" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "ds-idrepo-2" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "ds-cts-0" 21:03:48 DEBUG }, 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "ds-cts-2" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0.5715333333333338" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "ds-idrepo-0" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "ds-idrepo-2" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "0" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "3334.8333333333335" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643428" 21:03:48 INFO [http_cmd]: http status code OK 21:03:48 DEBUG --- status code --- 21:03:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:48 DEBUG --- http response --- 21:03:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:03:48 DEBUG ----- output ----- 21:03:48 DEBUG { 21:03:48 DEBUG "status": "success", 21:03:48 DEBUG "data": { 21:03:48 DEBUG "resultType": "vector", 21:03:48 DEBUG "result": [ 21:03:48 DEBUG { 21:03:48 DEBUG "metric": { 21:03:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG [...] 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG 21:03:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:48 DEBUG }, 21:03:48 DEBUG "value": [ 21:03:48 DEBUG 1713643428, 21:03:48 DEBUG "508.2545506500929" 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG ] 21:03:48 DEBUG } 21:03:48 DEBUG } 21:03:49 INFO 21:03:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:03:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:03:49 INFO [loop_until]: OK (rc = 0) 21:03:49 DEBUG --- stdout --- 21:03:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22160m 8679Mi am-767f6b67b5-qcqz2 22172m 8877Mi am-767f6b67b5-vbltz 23541m 9087Mi ds-cts-0 19452m 12218Mi ds-cts-1 19606m 12200Mi ds-cts-2 20228m 12332Mi ds-idrepo-0 9552m 3942Mi ds-idrepo-1 9306m 3927Mi ds-idrepo-2 9457m 3870Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 7m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11451m 4751Mi 21:03:49 DEBUG --- stderr --- 21:03:49 DEBUG 21:03:49 INFO 21:03:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:03:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:03:49 INFO [loop_until]: OK (rc = 0) 21:03:49 DEBUG --- stdout --- 21:03:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9949m 33% 7690Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21780m 72% 11402Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19596m 65% 15447Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 5171Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22167m 74% 11777Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19098m 63% 2983Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24548m 82% 11334Mi 10% gke-xlou-cdm-ds-0f601751-j488 10375m 34% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19574m 65% 17208Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10232m 34% 7901Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20166m 67% 15035Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11617m 73% 10248Mi 17% 21:03:49 DEBUG --- stderr --- 21:03:49 DEBUG 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "instance": "10.142.0.107:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "instance": "10.142.0.43:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0.07033333333333948" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 ERROR [http_cmd]: ERROR 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 ERROR --- status code --- 21:04:48 DEBUG }, 21:04:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:04:48 DEBUG "value": [ 21:04:48 ERROR --- http response --- 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:04:48 DEBUG 21:04:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "3329" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "ds-idrepo-0" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "ds-cts-2" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "ds-cts-0" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "ds-cts-2" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "28.266666666666666" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "307124509.164557" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "ds-cts-0" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "ds-cts-2" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "3330.9333333333334" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "5081137152" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG }, 21:04:48 DEBUG { 21:04:48 DEBUG "value": [ 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG 21:04:48 DEBUG "data": { 21:04:48 DEBUG 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG 21:04:48 DEBUG "result": [ 21:04:48 DEBUG [...] 21:04:48 DEBUG { 21:04:48 DEBUG 21:04:48 DEBUG "metric": { 21:04:48 DEBUG 21:04:48 DEBUG "pod": "ds-idrepo-0" 21:04:48 DEBUG 21:04:48 DEBUG }, 21:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:48 DEBUG "value": [ 21:04:48 DEBUG }, 21:04:48 DEBUG 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG 21:04:48 DEBUG "3320.2999999999997" 21:04:48 DEBUG [...] 21:04:48 DEBUG ] 21:04:48 DEBUG 21:04:48 DEBUG } 21:04:48 DEBUG 21:04:48 DEBUG ] 21:04:48 DEBUG 21:04:48 DEBUG } 21:04:48 DEBUG "pod": "ds-idrepo-2" 21:04:48 DEBUG } 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "instance": "10.142.0.107:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "instance": "10.142.0.43:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "3870078958592" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "instance": "10.142.0.107:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "instance": "10.142.0.43:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0.36973333333343134" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "instance": "10.142.0.107:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "instance": "10.142.0.43:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "100305722.53333333" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "599796" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "ds-cts-0" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "ds-cts-2" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0.5902666666666656" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0.1605" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "11.363103929327815" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "instance": "10.142.0.107:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "instance": "10.142.0.43:9100" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "100309181.63333333" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "22.047534312237495" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "ds-idrepo-0" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "ds-idrepo-2" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "ds-idrepo-0" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "ds-idrepo-2" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "0" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "3331.5" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643488" 21:04:48 INFO [http_cmd]: http status code OK 21:04:48 DEBUG --- status code --- 21:04:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:48 DEBUG --- http response --- 21:04:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:04:48 DEBUG ----- output ----- 21:04:48 DEBUG { 21:04:48 DEBUG "status": "success", 21:04:48 DEBUG "data": { 21:04:48 DEBUG "resultType": "vector", 21:04:48 DEBUG "result": [ 21:04:48 DEBUG { 21:04:48 DEBUG "metric": { 21:04:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG [...] 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG 21:04:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:48 DEBUG }, 21:04:48 DEBUG "value": [ 21:04:48 DEBUG 1713643488, 21:04:48 DEBUG "2765.232067510549" 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG ] 21:04:48 DEBUG } 21:04:48 DEBUG } 21:04:49 INFO 21:04:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:04:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:04:49 INFO [loop_until]: OK (rc = 0) 21:04:49 DEBUG --- stdout --- 21:04:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22386m 8676Mi am-767f6b67b5-qcqz2 22002m 8877Mi am-767f6b67b5-vbltz 23672m 9089Mi ds-cts-0 19836m 12216Mi ds-cts-1 19590m 12205Mi ds-cts-2 20463m 12334Mi ds-idrepo-0 9255m 3941Mi ds-idrepo-1 9516m 3928Mi ds-idrepo-2 9581m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11567m 4752Mi 21:04:49 DEBUG --- stderr --- 21:04:49 DEBUG 21:04:49 INFO 21:04:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:04:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:04:49 INFO [loop_until]: OK (rc = 0) 21:04:49 DEBUG --- stdout --- 21:04:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6292Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9939m 33% 7689Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21136m 70% 11402Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3351Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19895m 66% 15420Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5187Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22209m 74% 11774Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19751m 66% 2979Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 23857m 79% 11322Mi 10% gke-xlou-cdm-ds-0f601751-j488 10399m 34% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19789m 66% 17213Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10171m 34% 7895Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20221m 67% 15027Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11610m 73% 10259Mi 17% 21:04:49 DEBUG --- stderr --- 21:04:49 DEBUG 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "instance": "10.142.0.107:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "instance": "10.142.0.43:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0.08233333333333613" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "ds-cts-0" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "ds-cts-2" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "27.066666666666666" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643548" 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG { 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "ds-idrepo-0" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "ds-cts-2" 21:05:48 DEBUG }, 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG "value": [ 21:05:48 ERROR [http_cmd]: ERROR 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG 1713643548, 21:05:48 ERROR --- status code --- 21:05:48 DEBUG { 21:05:48 DEBUG "0" 21:05:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG ] 21:05:48 ERROR --- http response --- 21:05:48 DEBUG "data": { 21:05:48 DEBUG } 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG ] 21:05:48 DEBUG "result": [ 21:05:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG } 21:05:48 DEBUG { 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG } 21:05:48 DEBUG "metric": { 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG }, 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG "value": [ 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG { 21:05:48 DEBUG 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG 21:05:48 DEBUG "data": { 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG [...] 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "data": { 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "result": [ 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG { 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG "metric": { 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:48 DEBUG "0" 21:05:48 DEBUG 21:05:48 DEBUG }, 21:05:48 DEBUG ] 21:05:48 DEBUG [...] 21:05:48 DEBUG "value": [ 21:05:48 DEBUG } 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG ] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG } 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG } 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG [...] 21:05:48 DEBUG }, 21:05:48 DEBUG 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG 21:05:48 DEBUG "0" 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG ] 21:05:48 DEBUG }, 21:05:48 DEBUG } 21:05:48 DEBUG "value": [ 21:05:48 DEBUG ] 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG } 21:05:48 DEBUG "3329.866666666667" 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "5084221440" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG { 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "3327.8333333333335" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "ds-cts-0" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "ds-cts-2" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "281773.99479133583" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "ds-idrepo-0" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "ds-idrepo-2" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "3314.133333333333" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "instance": "10.142.0.107:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "instance": "10.142.0.43:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "3875240991744" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "instance": "10.142.0.107:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "instance": "10.142.0.43:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0.40059999999998014" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "600436.5" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "instance": "10.142.0.107:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "instance": "10.142.0.43:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "101292334.63333333" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "instance": "10.142.0.107:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "instance": "10.142.0.43:9100" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "101296279.23333332" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "ds-cts-0" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "ds-cts-2" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0.5183000000000011" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0.8985" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "11.39182763160475" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "ds-idrepo-0" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "ds-idrepo-2" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "24.434514101505506" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "ds-idrepo-0" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "ds-idrepo-2" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "0" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643548" 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "3329.5666666666666" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:48 INFO [http_cmd]: http status code OK 21:05:48 DEBUG --- status code --- 21:05:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:48 DEBUG --- http response --- 21:05:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:48 DEBUG ----- output ----- 21:05:48 DEBUG { 21:05:48 DEBUG "status": "success", 21:05:48 DEBUG "data": { 21:05:48 DEBUG "resultType": "vector", 21:05:48 DEBUG "result": [ 21:05:48 DEBUG { 21:05:48 DEBUG "metric": { 21:05:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG [...] 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG 21:05:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:48 DEBUG }, 21:05:48 DEBUG "value": [ 21:05:48 DEBUG 1713643548, 21:05:48 DEBUG "3122.149526773804" 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG ] 21:05:48 DEBUG } 21:05:48 DEBUG } 21:05:49 INFO 21:05:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:05:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:05:49 INFO [loop_until]: OK (rc = 0) 21:05:49 DEBUG --- stdout --- 21:05:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22699m 8710Mi am-767f6b67b5-qcqz2 21972m 8917Mi am-767f6b67b5-vbltz 24636m 9103Mi ds-cts-0 19792m 12217Mi ds-cts-1 19216m 12201Mi ds-cts-2 20618m 12337Mi ds-idrepo-0 9275m 3942Mi ds-idrepo-1 8970m 3934Mi ds-idrepo-2 9799m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11364m 4758Mi 21:05:49 DEBUG --- stderr --- 21:05:49 DEBUG 21:05:49 INFO 21:05:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:05:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:05:49 INFO [loop_until]: OK (rc = 0) 21:05:49 DEBUG --- stdout --- 21:05:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9904m 33% 7686Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22175m 74% 11448Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3347Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19883m 66% 15447Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22660m 75% 11815Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19606m 65% 2996Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24882m 83% 11333Mi 10% gke-xlou-cdm-ds-0f601751-j488 10246m 34% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19496m 65% 17211Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10207m 34% 7896Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20554m 68% 15028Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11595m 72% 10248Mi 17% 21:05:49 DEBUG --- stderr --- 21:05:49 DEBUG 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "3321.7" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "instance": "10.142.0.107:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "instance": "10.142.0.43:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0.08933333333333356" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643608" 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "ds-cts-0" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "ds-cts-2" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "29.066666666666666" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "5096280064" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 ERROR [http_cmd]: ERROR 21:06:48 ERROR --- status code --- 21:06:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:06:48 ERROR --- http response --- 21:06:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:06:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "ds-idrepo-0" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "ds-cts-2" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "3340.6666666666665" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "315435078.62068963" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "ds-cts-0" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "ds-cts-2" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "3946.7666666666664" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "ds-idrepo-0" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "ds-idrepo-2" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "instance": "10.142.0.107:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "instance": "10.142.0.43:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "3880450188288" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "instance": "10.142.0.107:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "instance": "10.142.0.43:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0.4073666666666099" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "600082.5" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "instance": "10.142.0.107:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "instance": "10.142.0.43:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "100683378.53333333" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "ds-idrepo-0" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "ds-idrepo-2" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "instance": "10.142.0.107:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "instance": "10.142.0.43:9100" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "100686546.33333333" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "ds-cts-0" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "ds-cts-2" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0.5973333333333357" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "4.041" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "11.416728713848348" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "ds-idrepo-0" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "ds-idrepo-2" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "0" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "22.37136158620706" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "3322.133333333333" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643608" 21:06:48 INFO [http_cmd]: http status code OK 21:06:48 DEBUG --- status code --- 21:06:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:48 DEBUG --- http response --- 21:06:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:48 DEBUG ----- output ----- 21:06:48 DEBUG { 21:06:48 DEBUG "status": "success", 21:06:48 DEBUG "data": { 21:06:48 DEBUG "resultType": "vector", 21:06:48 DEBUG "result": [ 21:06:48 DEBUG { 21:06:48 DEBUG "metric": { 21:06:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG [...] 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG 21:06:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:48 DEBUG }, 21:06:48 DEBUG "value": [ 21:06:48 DEBUG 1713643608, 21:06:48 DEBUG "1129.9310344827586" 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG ] 21:06:48 DEBUG } 21:06:48 DEBUG } 21:06:49 INFO 21:06:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:06:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:06:49 INFO [loop_until]: OK (rc = 0) 21:06:49 DEBUG --- stdout --- 21:06:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21859m 8723Mi am-767f6b67b5-qcqz2 22114m 8917Mi am-767f6b67b5-vbltz 24235m 9111Mi ds-cts-0 19790m 12216Mi ds-cts-1 19883m 12209Mi ds-cts-2 20638m 12342Mi ds-idrepo-0 9502m 3941Mi ds-idrepo-1 9483m 3934Mi ds-idrepo-2 9718m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11365m 4769Mi 21:06:49 DEBUG --- stderr --- 21:06:49 DEBUG 21:06:49 INFO 21:06:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:06:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:06:49 INFO [loop_until]: OK (rc = 0) 21:06:49 DEBUG --- stdout --- 21:06:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9956m 33% 7689Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22188m 74% 11454Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3351Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19682m 65% 15449Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22319m 74% 11818Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19812m 66% 3005Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24861m 83% 11329Mi 10% gke-xlou-cdm-ds-0f601751-j488 10305m 34% 7280Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19550m 65% 17213Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10186m 34% 7900Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19900m 66% 15037Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11656m 73% 10255Mi 17% 21:06:49 DEBUG --- stderr --- 21:06:49 DEBUG 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 ERROR [http_cmd]: ERROR 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 ERROR --- status code --- 21:07:48 DEBUG --- status code --- 21:07:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 ERROR --- http response --- 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:07:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "instance": "10.142.0.107:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "instance": "10.142.0.43:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "0.08633333333333247" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "ds-idrepo-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:48 DEBUG 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG 21:07:48 DEBUG { 21:07:48 DEBUG 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "pod": "ds-cts-2" 21:07:48 DEBUG "data": { 21:07:48 DEBUG }, 21:07:48 DEBUG "resultType": "vector", 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 DEBUG "value": [ 21:07:48 DEBUG "result": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG { 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 DEBUG "0" 21:07:48 DEBUG "metric": { 21:07:48 DEBUG ] 21:07:48 DEBUG "pod": "ds-cts-0" 21:07:48 DEBUG } 21:07:48 DEBUG }, 21:07:48 DEBUG ] 21:07:48 DEBUG "value": [ 21:07:48 DEBUG } 21:07:48 DEBUG 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 DEBUG 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG [...] 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG 21:07:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:07:48 DEBUG 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG "pod": "ds-cts-2" 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG }, 21:07:48 DEBUG { 21:07:48 DEBUG "value": [ 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "data": { 21:07:48 DEBUG "30.5" 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG ] 21:07:48 DEBUG "result": [ 21:07:48 DEBUG } 21:07:48 DEBUG { 21:07:48 DEBUG ] 21:07:48 DEBUG "metric": { 21:07:48 DEBUG } 21:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:48 DEBUG } 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "3335.1666666666665" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-cts-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "48631513.43488765" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "0" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "ds-cts-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-cts-2" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "0" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "5098807296" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-cts-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "428498262.11747116" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "3341.766666666667" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "3989.3999999999996" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "ds-idrepo-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-idrepo-2" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "0" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "instance": "10.142.0.107:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "instance": "10.142.0.43:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "3885596097536" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "instance": "10.142.0.107:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "instance": "10.142.0.43:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "0.39733333333324766" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643668" 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:48 DEBUG 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:48 DEBUG { 21:07:48 DEBUG }, 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "value": [ 21:07:48 DEBUG "data": { 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "601107" 21:07:48 DEBUG "result": [ 21:07:48 DEBUG ] 21:07:48 DEBUG { 21:07:48 DEBUG } 21:07:48 DEBUG "metric": { 21:07:48 DEBUG ] 21:07:48 DEBUG "pod": "ds-cts-0" 21:07:48 DEBUG } 21:07:48 DEBUG }, 21:07:48 DEBUG } 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-cts-2" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "0.6206999999999994" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG { 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG "metric": { 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG "pod": "ds-idrepo-0" 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:48 DEBUG 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG 21:07:48 DEBUG { 21:07:48 DEBUG 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG [...] 21:07:48 DEBUG "data": { 21:07:48 DEBUG 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG 21:07:48 DEBUG "result": [ 21:07:48 DEBUG 21:07:48 DEBUG { 21:07:48 DEBUG "pod": "ds-idrepo-2" 21:07:48 DEBUG "metric": { 21:07:48 DEBUG }, 21:07:48 DEBUG "instance": "10.142.0.107:9100" 21:07:48 DEBUG "value": [ 21:07:48 DEBUG }, 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG "0" 21:07:48 DEBUG 21:07:48 DEBUG ] 21:07:48 DEBUG 21:07:48 DEBUG } 21:07:48 DEBUG 21:07:48 DEBUG ] 21:07:48 DEBUG [...] 21:07:48 DEBUG } 21:07:48 DEBUG 21:07:48 DEBUG } 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "instance": "10.142.0.43:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "100963765.96666667" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "instance": "10.142.0.107:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "instance": "10.142.0.43:9100" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "100966798.86666667" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "8.1" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "11.45670609241836" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "ds-idrepo-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-idrepo-2" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "0" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-cts-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "14.724479359211902" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "3332.0333333333333" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643668" 21:07:48 INFO [http_cmd]: http status code OK 21:07:48 DEBUG --- status code --- 21:07:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:48 DEBUG --- http response --- 21:07:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:07:48 DEBUG ----- output ----- 21:07:48 DEBUG { 21:07:48 DEBUG "status": "success", 21:07:48 DEBUG "data": { 21:07:48 DEBUG "resultType": "vector", 21:07:48 DEBUG "result": [ 21:07:48 DEBUG { 21:07:48 DEBUG "metric": { 21:07:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG [...] 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG 21:07:48 DEBUG "pod": "ds-cts-0" 21:07:48 DEBUG }, 21:07:48 DEBUG "value": [ 21:07:48 DEBUG 1713643668, 21:07:48 DEBUG "58919723.64333835" 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG ] 21:07:48 DEBUG } 21:07:48 DEBUG } 21:07:49 INFO 21:07:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:07:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:07:49 INFO [loop_until]: OK (rc = 0) 21:07:49 DEBUG --- stdout --- 21:07:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22705m 8723Mi am-767f6b67b5-qcqz2 22097m 8956Mi am-767f6b67b5-vbltz 23635m 9117Mi ds-cts-0 19643m 12217Mi ds-cts-1 18833m 12226Mi ds-cts-2 20126m 12377Mi ds-idrepo-0 9799m 3941Mi ds-idrepo-1 9281m 3950Mi ds-idrepo-2 9794m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11254m 4772Mi 21:07:49 DEBUG --- stderr --- 21:07:49 DEBUG 21:07:49 INFO 21:07:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:07:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:07:49 INFO [loop_until]: OK (rc = 0) 21:07:49 DEBUG --- stdout --- 21:07:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9973m 33% 7702Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22351m 74% 11460Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3346Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20010m 67% 15433Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5181Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22646m 75% 11862Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19875m 66% 3004Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24999m 83% 11353Mi 10% gke-xlou-cdm-ds-0f601751-j488 10327m 34% 7280Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19489m 65% 17225Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10190m 34% 7896Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20420m 68% 15054Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11688m 73% 10253Mi 17% 21:07:49 DEBUG --- stderr --- 21:07:49 DEBUG 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "instance": "10.142.0.107:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "instance": "10.142.0.43:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0.06333333333334962" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 ERROR [http_cmd]: ERROR 21:08:48 ERROR --- status code --- 21:08:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:08:48 ERROR --- http response --- 21:08:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:08:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "3332.7" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG { 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "3348.2999999999997" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "ds-idrepo-0" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "ds-cts-2" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "ds-cts-0" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "ds-cts-2" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "29.966666666666665" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "ds-cts-0" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "ds-cts-2" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "5102485504" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "305989.85573459545" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "3366.1" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "ds-idrepo-0" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "ds-idrepo-2" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "instance": "10.142.0.107:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "instance": "10.142.0.43:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "3890647526400" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "instance": "10.142.0.107:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "instance": "10.142.0.43:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0.35266666666666424" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "ds-cts-0" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "ds-cts-2" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0.5784666666666661" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "600315" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "ds-idrepo-0" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "ds-idrepo-2" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "instance": "10.142.0.107:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "instance": "10.142.0.43:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "100876329.06666666" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "instance": "10.142.0.107:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "instance": "10.142.0.43:9100" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "100880294.13333333" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "10.908" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "11.329687459652277" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "ds-idrepo-0" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "ds-idrepo-2" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "3347.9" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "22.564660850936296" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643728" 21:08:48 INFO [http_cmd]: http status code OK 21:08:48 DEBUG --- status code --- 21:08:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:48 DEBUG --- http response --- 21:08:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:48 DEBUG ----- output ----- 21:08:48 DEBUG { 21:08:48 DEBUG "status": "success", 21:08:48 DEBUG "data": { 21:08:48 DEBUG "resultType": "vector", 21:08:48 DEBUG "result": [ 21:08:48 DEBUG { 21:08:48 DEBUG "metric": { 21:08:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG [...] 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG 21:08:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:08:48 DEBUG }, 21:08:48 DEBUG "value": [ 21:08:48 DEBUG 1713643728, 21:08:48 DEBUG "0" 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG ] 21:08:48 DEBUG } 21:08:48 DEBUG } 21:08:49 INFO 21:08:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:08:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:08:49 INFO [loop_until]: OK (rc = 0) 21:08:49 DEBUG --- stdout --- 21:08:49 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21832m 8722Mi am-767f6b67b5-qcqz2 22214m 8928Mi am-767f6b67b5-vbltz 24809m 9116Mi ds-cts-0 19596m 12217Mi ds-cts-1 20667m 12220Mi ds-cts-2 20266m 12374Mi ds-idrepo-0 9407m 3942Mi ds-idrepo-1 9433m 3950Mi ds-idrepo-2 9702m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11132m 4775Mi 21:08:49 DEBUG --- stderr --- 21:08:49 DEBUG 21:08:49 INFO 21:08:49 INFO [loop_until]: kubectl --namespace=xlou top node 21:08:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:08:50 INFO [loop_until]: OK (rc = 0) 21:08:50 DEBUG --- stdout --- 21:08:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9955m 33% 7708Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22141m 74% 11460Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3347Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19684m 65% 15438Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22642m 75% 11838Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19690m 65% 3002Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24772m 82% 11346Mi 10% gke-xlou-cdm-ds-0f601751-j488 10358m 34% 7278Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19788m 66% 17231Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10198m 34% 7927Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20444m 68% 15068Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11557m 72% 10255Mi 17% 21:08:50 DEBUG --- stderr --- 21:08:50 DEBUG 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "instance": "10.142.0.107:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "instance": "10.142.0.43:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0.08966666666666277" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "3343.4333333333334" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 ERROR [http_cmd]: ERROR 21:09:48 ERROR --- status code --- 21:09:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:09:48 ERROR --- http response --- 21:09:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:09:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "ds-cts-0" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "ds-cts-2" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "28" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:48 DEBUG 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG { 21:09:48 DEBUG 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG [...] 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG "data": { 21:09:48 DEBUG 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG 21:09:48 DEBUG "result": [ 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG 21:09:48 DEBUG { 21:09:48 DEBUG { 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG }, 21:09:48 DEBUG "pod": "ds-idrepo-0" 21:09:48 DEBUG "data": { 21:09:48 DEBUG "value": [ 21:09:48 DEBUG }, 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG "result": [ 21:09:48 DEBUG "0" 21:09:48 DEBUG 21:09:48 DEBUG { 21:09:48 DEBUG ] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG "metric": { 21:09:48 DEBUG [...] 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG ] 21:09:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:09:48 DEBUG 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG } 21:09:48 DEBUG }, 21:09:48 DEBUG 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG } 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG 21:09:48 DEBUG "pod": "ds-cts-2" 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG }, 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG "value": [ 21:09:48 DEBUG { 21:09:48 DEBUG [...] 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG "0" 21:09:48 DEBUG "data": { 21:09:48 DEBUG 21:09:48 DEBUG ] 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG "result": [ 21:09:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:09:48 DEBUG ] 21:09:48 DEBUG { 21:09:48 DEBUG { 21:09:48 DEBUG }, 21:09:48 DEBUG } 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "value": [ 21:09:48 DEBUG } 21:09:48 DEBUG "data": { 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG }, 21:09:48 DEBUG "5201997824" 21:09:48 DEBUG "result": [ 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG "value": [ 21:09:48 DEBUG ] 21:09:48 DEBUG { 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG "metric": { 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG 21:09:48 DEBUG ] 21:09:48 DEBUG "pod": "ds-cts-0" 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG }, 21:09:48 DEBUG { 21:09:48 DEBUG [...] 21:09:48 DEBUG } 21:09:48 DEBUG "value": [ 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "data": { 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "result": [ 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG [...] 21:09:48 DEBUG { 21:09:48 DEBUG }, 21:09:48 DEBUG 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG 21:09:48 DEBUG }, 21:09:48 DEBUG "3317.9" 21:09:48 DEBUG "pod": "ds-cts-2" 21:09:48 DEBUG "value": [ 21:09:48 DEBUG ] 21:09:48 DEBUG }, 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG ] 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG "0" 21:09:48 DEBUG [...] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG 21:09:48 DEBUG ] 21:09:48 DEBUG 21:09:48 DEBUG } 21:09:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:09:48 DEBUG } 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "3991.133333333333" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "ds-idrepo-0" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "ds-idrepo-2" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "instance": "10.142.0.107:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "instance": "10.142.0.43:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "3895684373504" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "instance": "10.142.0.107:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "instance": "10.142.0.43:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0.41536666666660366" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "instance": "10.142.0.107:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "instance": "10.142.0.43:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "101481068.06666666" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "599823.5" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "ds-cts-0" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "ds-cts-2" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0.5628999999999982" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "ds-idrepo-0" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "ds-idrepo-2" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "instance": "10.142.0.107:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "instance": "10.142.0.43:9100" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "101484023.06666666" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "3343.3333333333335" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "13.503" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "11.35649627737876" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "ds-idrepo-0" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "ds-idrepo-2" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643788" 21:09:48 INFO [http_cmd]: http status code OK 21:09:48 DEBUG --- status code --- 21:09:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:48 DEBUG --- http response --- 21:09:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:48 DEBUG ----- output ----- 21:09:48 DEBUG { 21:09:48 DEBUG "status": "success", 21:09:48 DEBUG "data": { 21:09:48 DEBUG "resultType": "vector", 21:09:48 DEBUG "result": [ 21:09:48 DEBUG { 21:09:48 DEBUG "metric": { 21:09:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG [...] 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG 21:09:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:09:48 DEBUG }, 21:09:48 DEBUG "value": [ 21:09:48 DEBUG 1713643788, 21:09:48 DEBUG "0" 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG ] 21:09:48 DEBUG } 21:09:48 DEBUG } 21:09:49 INFO 21:09:49 INFO [loop_until]: kubectl --namespace=xlou top pods 21:09:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:09:50 INFO [loop_until]: OK (rc = 0) 21:09:50 DEBUG --- stdout --- 21:09:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22246m 8833Mi am-767f6b67b5-qcqz2 22318m 8928Mi am-767f6b67b5-vbltz 24735m 9103Mi ds-cts-0 20616m 12212Mi ds-cts-1 19255m 12226Mi ds-cts-2 20797m 12379Mi ds-idrepo-0 9212m 3942Mi ds-idrepo-1 9195m 3951Mi ds-idrepo-2 9643m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11117m 4869Mi 21:09:50 DEBUG --- stderr --- 21:09:50 DEBUG 21:09:50 INFO 21:09:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:09:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:09:50 INFO [loop_until]: OK (rc = 0) 21:09:50 DEBUG --- stdout --- 21:09:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9881m 33% 7701Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22511m 75% 11575Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 94m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19320m 64% 15439Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22513m 75% 11833Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19752m 66% 3014Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24986m 83% 11347Mi 10% gke-xlou-cdm-ds-0f601751-j488 10317m 34% 7284Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19505m 65% 17228Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10199m 34% 7927Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20576m 68% 15074Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11224m 70% 10346Mi 17% 21:09:50 DEBUG --- stderr --- 21:09:50 DEBUG 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "instance": "10.142.0.107:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "instance": "10.142.0.43:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0.07799999999998969" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "3323.4333333333334" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 ERROR [http_cmd]: ERROR 21:10:48 ERROR --- status code --- 21:10:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:10:48 ERROR --- http response --- 21:10:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:10:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "ds-cts-0" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-cts-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "30.233333333333334" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "3336.8333333333335" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "ds-cts-0" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-cts-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-cts-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "ds-idrepo-0" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-cts-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "5202886656" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "3665.2333333333336" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "instance": "10.142.0.107:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "instance": "10.142.0.43:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "3900756962304" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "ds-idrepo-0" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-idrepo-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "instance": "10.142.0.107:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "instance": "10.142.0.43:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0.3849666666666688" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG { 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "ds-idrepo-0" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG 21:10:48 DEBUG { 21:10:48 DEBUG 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG [...] 21:10:48 DEBUG "data": { 21:10:48 DEBUG 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG 21:10:48 DEBUG "result": [ 21:10:48 DEBUG 21:10:48 DEBUG { 21:10:48 DEBUG "pod": "ds-idrepo-2" 21:10:48 DEBUG "metric": { 21:10:48 DEBUG }, 21:10:48 DEBUG "instance": "10.142.0.107:9100" 21:10:48 DEBUG "value": [ 21:10:48 DEBUG }, 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG "0" 21:10:48 DEBUG 21:10:48 DEBUG ] 21:10:48 DEBUG 21:10:48 DEBUG } 21:10:48 DEBUG 21:10:48 DEBUG ] 21:10:48 DEBUG [...] 21:10:48 DEBUG } 21:10:48 DEBUG 21:10:48 DEBUG } 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "instance": "10.142.0.43:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "100342945.96666667" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "ds-cts-0" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-cts-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0.5448666666666668" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "instance": "10.142.0.107:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "instance": "10.142.0.43:9100" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "100347558.76666667" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "599168.5" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "11.366547205933657" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "3321.9333333333334" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "17.915" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643848" 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-cts-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:48 INFO [http_cmd]: http status code OK 21:10:48 DEBUG --- status code --- 21:10:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:48 DEBUG --- http response --- 21:10:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:48 DEBUG ----- output ----- 21:10:48 DEBUG { 21:10:48 DEBUG "status": "success", 21:10:48 DEBUG "data": { 21:10:48 DEBUG "resultType": "vector", 21:10:48 DEBUG "result": [ 21:10:48 DEBUG { 21:10:48 DEBUG "metric": { 21:10:48 DEBUG "pod": "ds-idrepo-0" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG [...] 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG 21:10:48 DEBUG "pod": "ds-idrepo-2" 21:10:48 DEBUG }, 21:10:48 DEBUG "value": [ 21:10:48 DEBUG 1713643848, 21:10:48 DEBUG "0" 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG ] 21:10:48 DEBUG } 21:10:48 DEBUG } 21:10:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643849" 21:10:49 INFO [http_cmd]: http status code OK 21:10:49 DEBUG --- status code --- 21:10:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:49 DEBUG --- http response --- 21:10:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:49 DEBUG ----- output ----- 21:10:49 DEBUG { 21:10:49 DEBUG "status": "success", 21:10:49 DEBUG "data": { 21:10:49 DEBUG "resultType": "vector", 21:10:49 DEBUG "result": [ 21:10:49 DEBUG { 21:10:49 DEBUG "metric": { 21:10:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:10:49 DEBUG }, 21:10:49 DEBUG "value": [ 21:10:49 DEBUG 21:10:49 DEBUG 21:10:49 DEBUG 21:10:49 DEBUG [...] 21:10:49 DEBUG 21:10:49 DEBUG 21:10:49 DEBUG 21:10:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:10:49 DEBUG }, 21:10:49 DEBUG "value": [ 21:10:49 DEBUG 1713643849, 21:10:49 DEBUG "0" 21:10:49 DEBUG ] 21:10:49 DEBUG } 21:10:49 DEBUG ] 21:10:49 DEBUG } 21:10:49 DEBUG } 21:10:50 INFO 21:10:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:10:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:10:50 INFO [loop_until]: OK (rc = 0) 21:10:50 DEBUG --- stdout --- 21:10:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22843m 8808Mi am-767f6b67b5-qcqz2 22112m 8925Mi am-767f6b67b5-vbltz 24224m 9102Mi ds-cts-0 20014m 12217Mi ds-cts-1 19867m 12228Mi ds-cts-2 21008m 12376Mi ds-idrepo-0 9304m 3941Mi ds-idrepo-1 9668m 3950Mi ds-idrepo-2 9731m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11308m 4870Mi 21:10:50 DEBUG --- stderr --- 21:10:50 DEBUG 21:10:50 INFO 21:10:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:10:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:10:50 INFO [loop_until]: OK (rc = 0) 21:10:50 DEBUG --- stdout --- 21:10:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9933m 33% 7703Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22151m 74% 11550Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 97m 0% 3366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19878m 66% 15423Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22776m 76% 11841Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19714m 66% 2991Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24801m 83% 11340Mi 10% gke-xlou-cdm-ds-0f601751-j488 10370m 34% 7281Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19591m 65% 17238Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 9819m 32% 7923Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20650m 69% 15080Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11630m 73% 10348Mi 17% 21:10:50 DEBUG --- stderr --- 21:10:50 DEBUG 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "instance": "10.142.0.107:9100" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "instance": "10.142.0.43:9100" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "0.07199999999999704" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "3331.0333333333333" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 ERROR [http_cmd]: ERROR 21:11:48 ERROR --- status code --- 21:11:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:11:48 ERROR --- http response --- 21:11:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:11:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "3334" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "ds-cts-0" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "ds-cts-2" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "30.8" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "ds-cts-0" 21:11:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:48 DEBUG }, 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "ds-cts-2" 21:11:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:48 DEBUG }, 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "0" 21:11:48 DEBUG "0" 21:11:48 DEBUG ] 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "ds-cts-0" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "399624097.323601" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "ds-idrepo-0" 21:11:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:11:48 DEBUG }, 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "ds-cts-2" 21:11:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:11:48 DEBUG }, 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "0" 21:11:48 DEBUG "5204025344" 21:11:48 DEBUG ] 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "ds-cts-0" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "68527953.08503339" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "4014.133333333333" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "instance": "10.142.0.107:9100" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "instance": "10.142.0.43:9100" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "3905702321152" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "ds-idrepo-0" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "ds-idrepo-2" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "0" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "instance": "10.142.0.107:9100" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "instance": "10.142.0.43:9100" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "0.3671666666667268" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG { 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "ds-idrepo-0" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:48 DEBUG 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG 21:11:48 DEBUG { 21:11:48 DEBUG [...] 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG 21:11:48 DEBUG "data": { 21:11:48 DEBUG 21:11:48 DEBUG "resultType": "vector", 21:11:48 DEBUG 21:11:48 DEBUG "result": [ 21:11:48 DEBUG "pod": "ds-idrepo-2" 21:11:48 DEBUG { 21:11:48 DEBUG }, 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "value": [ 21:11:48 DEBUG "instance": "10.142.0.107:9100" 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG }, 21:11:48 DEBUG "0" 21:11:48 DEBUG "value": [ 21:11:48 DEBUG ] 21:11:48 DEBUG 21:11:48 DEBUG } 21:11:48 DEBUG 21:11:48 DEBUG ] 21:11:48 DEBUG 21:11:48 DEBUG } 21:11:48 DEBUG [...] 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643908" 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "instance": "10.142.0.43:9100" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "101396142.6" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: http status code OK 21:11:48 DEBUG --- status code --- 21:11:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:48 DEBUG --- http response --- 21:11:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:48 DEBUG ----- output ----- 21:11:48 DEBUG { 21:11:48 DEBUG "status": "success", 21:11:48 DEBUG "data": { 21:11:48 DEBUG "resultType": "vector", 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:48 DEBUG "result": [ 21:11:48 DEBUG { 21:11:48 DEBUG "metric": { 21:11:48 DEBUG "pod": "ds-cts-0" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG [...] 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG 21:11:48 DEBUG "pod": "ds-cts-2" 21:11:48 DEBUG }, 21:11:48 DEBUG "value": [ 21:11:48 DEBUG 1713643908, 21:11:48 DEBUG "0.6279666666666686" 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG ] 21:11:48 DEBUG } 21:11:48 DEBUG } 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643908" 21:11:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643908" 21:11:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643909" 21:11:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643909" 21:11:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643909" 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 DEBUG --- status code --- 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG { 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG "data": { 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "result": [ 21:11:49 DEBUG { 21:11:49 DEBUG "metric": { 21:11:49 DEBUG "pod": "ds-idrepo-0" 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG [...] 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG "pod": "ds-idrepo-2" 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 1713643909, 21:11:49 DEBUG "0" 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 DEBUG --- status code --- 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG { 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG "data": { 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "result": [ 21:11:49 DEBUG { 21:11:49 DEBUG "metric": { 21:11:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG [...] 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG "pod": "ds-cts-0" 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 1713643909, 21:11:49 DEBUG "19.89802691996143" 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 DEBUG --- status code --- 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 DEBUG --- status code --- 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG --- status code --- 21:11:49 DEBUG --- status code --- 21:11:49 DEBUG --- status code --- 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG { 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG { 21:11:49 DEBUG { 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG { 21:11:49 DEBUG { 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG "data": { 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG "data": { 21:11:49 DEBUG "data": { 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "data": { 21:11:49 DEBUG "data": { 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "result": [ 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "result": [ 21:11:49 DEBUG "result": [ 21:11:49 DEBUG { 21:11:49 DEBUG "result": [ 21:11:49 DEBUG "result": [ 21:11:49 DEBUG { 21:11:49 DEBUG { 21:11:49 DEBUG "metric": { 21:11:49 DEBUG { 21:11:49 DEBUG { 21:11:49 DEBUG "metric": { 21:11:49 DEBUG "metric": { 21:11:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:49 DEBUG "metric": { 21:11:49 DEBUG "metric": { 21:11:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:49 DEBUG }, 21:11:49 DEBUG "instance": "10.142.0.107:9100" 21:11:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:11:49 DEBUG }, 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG }, 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 21:11:49 DEBUG "value": [ 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG [...] 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG [...] 21:11:49 DEBUG [...] 21:11:49 DEBUG 21:11:49 DEBUG [...] 21:11:49 DEBUG [...] 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:49 DEBUG }, 21:11:49 DEBUG "instance": "10.142.0.43:9100" 21:11:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:11:49 DEBUG }, 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG }, 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 1713643908, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 1713643909, 21:11:49 DEBUG 1713643908, 21:11:49 DEBUG "599424" 21:11:49 DEBUG 1713643908, 21:11:49 DEBUG 1713643908, 21:11:49 DEBUG "3328.4" 21:11:49 DEBUG "20.049500000000002" 21:11:49 DEBUG ] 21:11:49 DEBUG "101400698.16666666" 21:11:49 DEBUG "11.42120128285652" 21:11:49 DEBUG ] 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG ] 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG ] 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG ] 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643909" 21:11:49 INFO [http_cmd]: http status code OK 21:11:49 DEBUG --- status code --- 21:11:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:49 DEBUG --- http response --- 21:11:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:49 DEBUG ----- output ----- 21:11:49 DEBUG { 21:11:49 DEBUG "status": "success", 21:11:49 DEBUG "data": { 21:11:49 DEBUG "resultType": "vector", 21:11:49 DEBUG "result": [ 21:11:49 DEBUG { 21:11:49 DEBUG "metric": { 21:11:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG [...] 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG 21:11:49 DEBUG "pod": "ds-cts-0" 21:11:49 DEBUG }, 21:11:49 DEBUG "value": [ 21:11:49 DEBUG 1713643909, 21:11:49 DEBUG "82446533.65774535" 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG ] 21:11:49 DEBUG } 21:11:49 DEBUG } 21:11:50 INFO 21:11:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:11:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:11:50 INFO [loop_until]: OK (rc = 0) 21:11:50 DEBUG --- stdout --- 21:11:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21870m 8810Mi am-767f6b67b5-qcqz2 22624m 8930Mi am-767f6b67b5-vbltz 24469m 9115Mi ds-cts-0 19532m 12215Mi ds-cts-1 19638m 12228Mi ds-cts-2 20658m 12374Mi ds-idrepo-0 9349m 3942Mi ds-idrepo-1 9465m 3950Mi ds-idrepo-2 9301m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11404m 4871Mi 21:11:50 DEBUG --- stderr --- 21:11:50 DEBUG 21:11:50 INFO 21:11:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:11:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:11:50 INFO [loop_until]: OK (rc = 0) 21:11:50 DEBUG --- stdout --- 21:11:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9902m 33% 7689Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22196m 74% 11545Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3349Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19839m 66% 15443Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5197Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22789m 76% 11838Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19829m 66% 3017Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25108m 84% 11346Mi 10% gke-xlou-cdm-ds-0f601751-j488 10282m 34% 7267Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19720m 66% 17230Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10057m 33% 7925Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20428m 68% 15074Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11654m 73% 10346Mi 17% 21:11:50 DEBUG --- stderr --- 21:11:50 DEBUG 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "instance": "10.142.0.107:9100" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "instance": "10.142.0.43:9100" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "0.07633333333332264" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "3328" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "3325.2666666666664" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 ERROR [http_cmd]: ERROR 21:12:48 DEBUG --- status code --- 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 ERROR --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- status code --- 21:12:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 ERROR --- http response --- 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:12:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG { 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG { 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG "data": { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "result": [ 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG { 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG "result": [ 21:12:48 DEBUG "metric": { 21:12:48 DEBUG { 21:12:48 DEBUG { 21:12:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "metric": { 21:12:48 DEBUG }, 21:12:48 DEBUG "data": { 21:12:48 DEBUG "pod": "ds-cts-0" 21:12:48 DEBUG "value": [ 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG }, 21:12:48 DEBUG 21:12:48 DEBUG "result": [ 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG { 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "metric": { 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG "pod": "ds-cts-0" 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG }, 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG 21:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:48 DEBUG 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG 21:12:48 DEBUG }, 21:12:48 DEBUG 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG "pod": "ds-cts-2" 21:12:48 DEBUG "value": [ 21:12:48 DEBUG [...] 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG }, 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG 21:12:48 DEBUG "value": [ 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:48 DEBUG "0" 21:12:48 DEBUG 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG ] 21:12:48 DEBUG 21:12:48 DEBUG "0" 21:12:48 DEBUG { 21:12:48 DEBUG } 21:12:48 DEBUG "pod": "ds-cts-2" 21:12:48 DEBUG ] 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG ] 21:12:48 DEBUG }, 21:12:48 DEBUG } 21:12:48 DEBUG "data": { 21:12:48 DEBUG } 21:12:48 DEBUG "value": [ 21:12:48 DEBUG ] 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG } 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG } 21:12:48 DEBUG "result": [ 21:12:48 DEBUG "31.933333333333334" 21:12:48 DEBUG } 21:12:48 DEBUG { 21:12:48 DEBUG ] 21:12:48 DEBUG "metric": { 21:12:48 DEBUG } 21:12:48 DEBUG "pod": "ds-idrepo-0" 21:12:48 DEBUG ] 21:12:48 DEBUG }, 21:12:48 DEBUG } 21:12:48 DEBUG "value": [ 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG } 21:12:48 DEBUG 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [...] 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:48 DEBUG 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG 21:12:48 DEBUG { 21:12:48 DEBUG 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "pod": "ds-cts-2" 21:12:48 DEBUG "data": { 21:12:48 DEBUG }, 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "value": [ 21:12:48 DEBUG "result": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG { 21:12:48 DEBUG "0" 21:12:48 DEBUG "metric": { 21:12:48 DEBUG ] 21:12:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:48 DEBUG } 21:12:48 DEBUG }, 21:12:48 DEBUG ] 21:12:48 DEBUG "value": [ 21:12:48 DEBUG } 21:12:48 DEBUG 21:12:48 DEBUG } 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "0" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "314801590.45201284" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "5209640960" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "3998.7666666666664" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "instance": "10.142.0.107:9100" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "instance": "10.142.0.43:9100" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "3910657858560" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643968" 21:12:48 INFO [http_cmd]: http status code OK 21:12:48 DEBUG --- status code --- 21:12:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:48 DEBUG --- http response --- 21:12:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:48 DEBUG ----- output ----- 21:12:48 DEBUG { 21:12:48 DEBUG "status": "success", 21:12:48 DEBUG "data": { 21:12:48 DEBUG "resultType": "vector", 21:12:48 DEBUG "result": [ 21:12:48 DEBUG { 21:12:48 DEBUG "metric": { 21:12:48 DEBUG "pod": "ds-idrepo-0" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG [...] 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG 21:12:48 DEBUG "pod": "ds-idrepo-2" 21:12:48 DEBUG }, 21:12:48 DEBUG "value": [ 21:12:48 DEBUG 1713643968, 21:12:48 DEBUG "0" 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG ] 21:12:48 DEBUG } 21:12:48 DEBUG } 21:12:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643968" 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "instance": "10.142.0.107:9100" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "instance": "10.142.0.43:9100" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643968, 21:12:49 DEBUG "0.40210000000002993" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643969" 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG { 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "pod": "ds-idrepo-0" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "pod": "ds-idrepo-2" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "0" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:49 DEBUG } 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG } 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "instance": "10.142.0.107:9100" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "instance": "10.142.0.43:9100" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "101019378.96666667" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "pod": "ds-cts-0" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "pod": "ds-cts-2" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "0.5956999999999993" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "3324.1" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "pod": "ds-idrepo-0" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "pod": "ds-idrepo-2" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "0" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "11.378012489216049" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713643969" 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713643969" 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "599782.5" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG { 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG "status": "success", 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG "data": { 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "metric": { 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:49 DEBUG "data": { 21:12:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG }, 21:12:49 DEBUG { 21:12:49 DEBUG "result": [ 21:12:49 DEBUG "value": [ 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG { 21:12:49 DEBUG 21:12:49 DEBUG "data": { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "instance": "10.142.0.107:9100" 21:12:49 DEBUG 21:12:49 DEBUG "result": [ 21:12:49 DEBUG }, 21:12:49 DEBUG [...] 21:12:49 DEBUG { 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG "metric": { 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG }, 21:12:49 DEBUG 21:12:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:49 DEBUG "value": [ 21:12:49 DEBUG [...] 21:12:49 DEBUG }, 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "25.139" 21:12:49 DEBUG [...] 21:12:49 DEBUG "instance": "10.142.0.43:9100" 21:12:49 DEBUG ] 21:12:49 DEBUG 21:12:49 DEBUG }, 21:12:49 DEBUG } 21:12:49 DEBUG 21:12:49 DEBUG "value": [ 21:12:49 DEBUG ] 21:12:49 DEBUG 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG } 21:12:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:12:49 DEBUG "101021739.16666666" 21:12:49 DEBUG } 21:12:49 DEBUG }, 21:12:49 DEBUG ] 21:12:49 DEBUG "value": [ 21:12:49 DEBUG } 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG ] 21:12:49 DEBUG "11.435004380218208" 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713643969" 21:12:49 INFO [http_cmd]: http status code OK 21:12:49 DEBUG --- status code --- 21:12:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:49 DEBUG --- http response --- 21:12:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:12:49 DEBUG ----- output ----- 21:12:49 DEBUG { 21:12:49 DEBUG "status": "success", 21:12:49 DEBUG "data": { 21:12:49 DEBUG "resultType": "vector", 21:12:49 DEBUG "result": [ 21:12:49 DEBUG { 21:12:49 DEBUG "metric": { 21:12:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG [...] 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG 21:12:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:12:49 DEBUG }, 21:12:49 DEBUG "value": [ 21:12:49 DEBUG 1713643969, 21:12:49 DEBUG "0" 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG ] 21:12:49 DEBUG } 21:12:49 DEBUG } 21:12:50 INFO 21:12:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:12:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:12:50 INFO [loop_until]: OK (rc = 0) 21:12:50 DEBUG --- stdout --- 21:12:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21976m 8816Mi am-767f6b67b5-qcqz2 22225m 8931Mi am-767f6b67b5-vbltz 24266m 9103Mi ds-cts-0 19641m 12216Mi ds-cts-1 19826m 12225Mi ds-cts-2 20538m 12374Mi ds-idrepo-0 9331m 3941Mi ds-idrepo-1 9344m 3950Mi ds-idrepo-2 9871m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 11m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11337m 4874Mi 21:12:50 DEBUG --- stderr --- 21:12:50 DEBUG 21:12:50 INFO 21:12:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:12:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:12:50 INFO [loop_until]: OK (rc = 0) 21:12:50 DEBUG --- stdout --- 21:12:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6281Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9893m 33% 7705Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22135m 74% 11564Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19934m 66% 15438Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5196Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22611m 75% 11840Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19669m 65% 2999Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25007m 83% 11344Mi 10% gke-xlou-cdm-ds-0f601751-j488 10367m 34% 7284Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19581m 65% 17236Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10112m 33% 7925Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20475m 68% 15079Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11699m 73% 10346Mi 17% 21:12:50 DEBUG --- stderr --- 21:12:50 DEBUG 21:13:04 WARNING File /tmp/lodemon_data-20240420_204036/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [20/Apr/2024 21:13:04] "GET /monitoring/average?start_time=20240420_194349&stop_time=20240420_201103 HTTP/1.1" 200 - 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "instance": "10.142.0.107:9100" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "instance": "10.142.0.43:9100" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "0.10066666666666986" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "0" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644028" 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "0" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 ERROR [http_cmd]: ERROR 21:13:48 ERROR --- status code --- 21:13:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:13:48 ERROR --- http response --- 21:13:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:13:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "ds-cts-0" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "ds-cts-2" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "0" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "ds-idrepo-0" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "ds-cts-2" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "0" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "4678877184" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG { 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "data": { 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG "result": [ 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG { 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:13:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:48 DEBUG "pod": "ds-cts-0" 21:13:48 DEBUG }, 21:13:48 DEBUG }, 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG "value": [ 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG [...] 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:13:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:48 DEBUG "pod": "ds-cts-2" 21:13:48 DEBUG }, 21:13:48 DEBUG }, 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG "value": [ 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "19555.43073790634" 21:13:48 DEBUG "0" 21:13:48 DEBUG "28.53333333333333" 21:13:48 DEBUG ] 21:13:48 DEBUG ] 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG ] 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "15828395.703152426" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "ds-idrepo-0" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "ds-idrepo-2" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "0" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "4159.833333333333" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644028" 21:13:48 INFO [http_cmd]: http status code OK 21:13:48 DEBUG --- status code --- 21:13:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:48 DEBUG --- http response --- 21:13:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:48 DEBUG ----- output ----- 21:13:48 DEBUG { 21:13:48 DEBUG "status": "success", 21:13:48 DEBUG "data": { 21:13:48 DEBUG "resultType": "vector", 21:13:48 DEBUG "result": [ 21:13:48 DEBUG { 21:13:48 DEBUG "metric": { 21:13:48 DEBUG "instance": "10.142.0.107:9100" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG [...] 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG 21:13:48 DEBUG "instance": "10.142.0.43:9100" 21:13:48 DEBUG }, 21:13:48 DEBUG "value": [ 21:13:48 DEBUG 1713644028, 21:13:48 DEBUG "3914981274624" 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG ] 21:13:48 DEBUG } 21:13:48 DEBUG } 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644029" 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "instance": "10.142.0.107:9100" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "instance": "10.142.0.43:9100" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "0.4887000000000484" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "ds-cts-0" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "pod": "ds-cts-2" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "28.424466666666667" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644029" 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "ds-idrepo-0" 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:49 DEBUG }, 21:13:49 DEBUG "instance": "10.142.0.107:9100" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "pod": "ds-idrepo-2" 21:13:49 DEBUG 21:13:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:49 DEBUG }, 21:13:49 DEBUG "instance": "10.142.0.43:9100" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "0" 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "0" 21:13:49 DEBUG ] 21:13:49 DEBUG "34018239.86666667" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644029" 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "instance": "10.142.0.107:9100" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "instance": "10.142.0.43:9100" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "34024893.7" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG { 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "ds-idrepo-0" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "pod": "ds-idrepo-2" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "0" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "2.2920359371267955" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644029" 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "28.8295" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "510601" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "1.718111310244678" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644029" 21:13:49 INFO [http_cmd]: http status code OK 21:13:49 DEBUG --- status code --- 21:13:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:49 DEBUG --- http response --- 21:13:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:49 DEBUG ----- output ----- 21:13:49 DEBUG { 21:13:49 DEBUG "status": "success", 21:13:49 DEBUG "data": { 21:13:49 DEBUG "resultType": "vector", 21:13:49 DEBUG "result": [ 21:13:49 DEBUG { 21:13:49 DEBUG "metric": { 21:13:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG [...] 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG 21:13:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:13:49 DEBUG }, 21:13:49 DEBUG "value": [ 21:13:49 DEBUG 1713644029, 21:13:49 DEBUG "107842.44892227762" 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG ] 21:13:49 DEBUG } 21:13:49 DEBUG } 21:13:50 INFO 21:13:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:13:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:13:50 INFO [loop_until]: OK (rc = 0) 21:13:50 DEBUG --- stdout --- 21:13:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3626m 8814Mi am-767f6b67b5-qcqz2 3457m 8929Mi am-767f6b67b5-vbltz 3886m 9107Mi ds-cts-0 18937m 12203Mi ds-cts-1 17932m 12222Mi ds-cts-2 19240m 12364Mi ds-idrepo-0 936m 3942Mi ds-idrepo-1 930m 3932Mi ds-idrepo-2 915m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-597977695c-ld9s4 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 904m 4305Mi 21:13:50 DEBUG --- stderr --- 21:13:50 DEBUG 21:13:50 INFO 21:13:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:13:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:13:50 INFO [loop_until]: OK (rc = 0) 21:13:50 DEBUG --- stdout --- 21:13:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1006m 3% 7695Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3586m 12% 11612Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18703m 62% 15425Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3536m 11% 11882Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 86m 0% 2988Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 4149m 13% 11380Mi 10% gke-xlou-cdm-ds-0f601751-j488 1001m 3% 7284Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18042m 60% 17195Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 1021m 3% 7907Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18637m 62% 15046Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 2145m 13% 9779Mi 16% 21:13:50 DEBUG --- stderr --- 21:13:50 DEBUG 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644088" 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "instance": "10.142.0.107:9100" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "instance": "10.142.0.43:9100" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "0.07633333333332075" 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644088" 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "3510.2" 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 ERROR [http_cmd]: ERROR 21:14:48 ERROR --- status code --- 21:14:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:14:48 ERROR --- http response --- 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:14:48 DEBUG --- status code --- 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "pod": "ds-cts-2" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "65080737.164955325" 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG --- status code --- 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG { 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "data": { 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG "result": [ 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG { 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "pod": "ds-idrepo-0" 21:14:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:48 DEBUG }, 21:14:48 DEBUG }, 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG "value": [ 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG [...] 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "pod": "ds-cts-2" 21:14:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:48 DEBUG }, 21:14:48 DEBUG }, 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG "value": [ 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644088" 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "0" 21:14:48 DEBUG "0" 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG "3538.0333333333333" 21:14:48 DEBUG ] 21:14:48 DEBUG ] 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG ] 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "pod": "ds-cts-0" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "pod": "ds-cts-2" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "0" 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "6347309056" 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "pod": "ds-cts-0" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "pod": "ds-cts-2" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "29.466666666666665" 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 INFO [http_cmd]: http status code OK 21:14:48 DEBUG --- status code --- 21:14:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:48 DEBUG --- http response --- 21:14:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:14:48 DEBUG ----- output ----- 21:14:48 DEBUG { 21:14:48 DEBUG "status": "success", 21:14:48 DEBUG "data": { 21:14:48 DEBUG "resultType": "vector", 21:14:48 DEBUG "result": [ 21:14:48 DEBUG { 21:14:48 DEBUG "metric": { 21:14:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG [...] 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG 21:14:48 DEBUG "pod": "ds-cts-2" 21:14:48 DEBUG }, 21:14:48 DEBUG "value": [ 21:14:48 DEBUG 1713644088, 21:14:48 DEBUG "151572.2096279504" 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG ] 21:14:48 DEBUG } 21:14:48 DEBUG } 21:14:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644088" 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "ds-idrepo-0" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "ds-idrepo-2" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644088, 21:14:49 DEBUG "0" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "4069.333333333333" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "instance": "10.142.0.107:9100" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "instance": "10.142.0.43:9100" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "3919902034944" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "instance": "10.142.0.107:9100" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "instance": "10.142.0.43:9100" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "0.3822999999999913" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "ds-cts-0" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "ds-cts-2" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "0.5398999999999982" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644089" 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "ds-idrepo-0" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "ds-idrepo-2" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "0" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG [...] 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:49 DEBUG 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG }, 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "value": [ 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG "3511.2666666666664" 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG ] 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG } 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "instance": "10.142.0.107:9100" 21:14:49 DEBUG ] 21:14:49 DEBUG "instance": "10.142.0.107:9100" 21:14:49 DEBUG }, 21:14:49 DEBUG } 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG } 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "instance": "10.142.0.43:9100" 21:14:49 DEBUG "instance": "10.142.0.43:9100" 21:14:49 DEBUG }, 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "105350811.56666666" 21:14:49 DEBUG "105346598.73333332" 21:14:49 DEBUG ] 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG { 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "ds-idrepo-0" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "ds-idrepo-2" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:14:49 DEBUG "0" 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG ] 21:14:49 DEBUG { 21:14:49 DEBUG } 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG ] 21:14:49 DEBUG "data": { 21:14:49 DEBUG } 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG } 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "ds-cts-2" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "20.466546406192602" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:49 DEBUG }, 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:49 DEBUG }, 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "405126" 21:14:49 DEBUG "32.793" 21:14:49 DEBUG ] 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "12.026999300463396" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644089" 21:14:49 INFO [http_cmd]: http status code OK 21:14:49 DEBUG --- status code --- 21:14:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:49 DEBUG --- http response --- 21:14:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:14:49 DEBUG ----- output ----- 21:14:49 DEBUG { 21:14:49 DEBUG "status": "success", 21:14:49 DEBUG "data": { 21:14:49 DEBUG "resultType": "vector", 21:14:49 DEBUG "result": [ 21:14:49 DEBUG { 21:14:49 DEBUG "metric": { 21:14:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG [...] 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG 21:14:49 DEBUG "pod": "ds-cts-2" 21:14:49 DEBUG }, 21:14:49 DEBUG "value": [ 21:14:49 DEBUG 1713644089, 21:14:49 DEBUG "80972059.20789438" 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG ] 21:14:49 DEBUG } 21:14:49 DEBUG } 21:14:50 INFO 21:14:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:14:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:14:50 INFO [loop_until]: OK (rc = 0) 21:14:50 DEBUG --- stdout --- 21:14:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 23920m 8820Mi am-767f6b67b5-qcqz2 22935m 8957Mi am-767f6b67b5-vbltz 24207m 9187Mi ds-cts-0 19599m 12215Mi ds-cts-1 19142m 12232Mi ds-cts-2 20215m 12372Mi ds-idrepo-0 9382m 3942Mi ds-idrepo-1 10050m 3933Mi ds-idrepo-2 10256m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 12002m 5959Mi 21:14:50 DEBUG --- stderr --- 21:14:50 DEBUG 21:14:50 INFO 21:14:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:14:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:14:50 INFO [loop_until]: OK (rc = 0) 21:14:50 DEBUG --- stdout --- 21:14:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10577m 35% 7707Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 24009m 80% 11560Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19777m 66% 15427Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5202Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 23767m 79% 11831Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 24091m 80% 3202Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25507m 85% 11476Mi 10% gke-xlou-cdm-ds-0f601751-j488 10905m 36% 7288Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19574m 65% 17234Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10694m 35% 7911Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20439m 68% 15073Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 12239m 77% 11317Mi 19% 21:14:50 DEBUG --- stderr --- 21:14:50 DEBUG 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644148" 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG { 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG "data": { 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "result": [ 21:15:48 DEBUG { 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "instance": "10.142.0.107:9100" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG "instance": "10.142.0.43:9100" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "0.08166666666667197" 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644148" 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG { 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG "data": { 21:15:48 DEBUG { 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG "result": [ 21:15:48 DEBUG "data": { 21:15:48 DEBUG { 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "result": [ 21:15:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:48 DEBUG { 21:15:48 DEBUG }, 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "value": [ 21:15:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:48 DEBUG 21:15:48 DEBUG }, 21:15:48 DEBUG 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG 21:15:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG 21:15:48 DEBUG }, 21:15:48 DEBUG 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:48 DEBUG "value": [ 21:15:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG }, 21:15:48 DEBUG { 21:15:48 DEBUG "1876.5" 21:15:48 DEBUG "value": [ 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG ] 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "data": { 21:15:48 DEBUG } 21:15:48 DEBUG "1877.1333333333334" 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG ] 21:15:48 DEBUG ] 21:15:48 DEBUG "result": [ 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:48 DEBUG { 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG "metric": { 21:15:48 DEBUG } 21:15:48 DEBUG "pod": "ds-cts-0" 21:15:48 DEBUG } 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG "pod": "ds-cts-2" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "0" 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:48 ERROR [http_cmd]: ERROR 21:15:48 ERROR --- status code --- 21:15:48 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:15:48 ERROR --- http response --- 21:15:48 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:15:48 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG { 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG "data": { 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "result": [ 21:15:48 DEBUG { 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG "pod": "ds-idrepo-0" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "0" 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG { 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG "data": { 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "result": [ 21:15:48 DEBUG { 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "pod": "ds-idrepo-0" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG "pod": "ds-cts-2" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "0" 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644148" 21:15:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644148" 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG { 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG "data": { 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "result": [ 21:15:48 DEBUG { 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "0" 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG { 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG "data": { 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "result": [ 21:15:48 DEBUG { 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "0" 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:48 INFO [http_cmd]: http status code OK 21:15:48 DEBUG --- status code --- 21:15:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:48 DEBUG --- http response --- 21:15:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:48 DEBUG ----- output ----- 21:15:48 DEBUG { 21:15:48 DEBUG "status": "success", 21:15:48 DEBUG "data": { 21:15:48 DEBUG "resultType": "vector", 21:15:48 DEBUG "result": [ 21:15:48 DEBUG { 21:15:48 DEBUG "metric": { 21:15:48 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG [...] 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG 21:15:48 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:15:48 DEBUG }, 21:15:48 DEBUG "value": [ 21:15:48 DEBUG 1713644148, 21:15:48 DEBUG "10260164608" 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG ] 21:15:48 DEBUG } 21:15:48 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "ds-cts-0" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "ds-cts-2" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "32.6" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "4159.366666666667" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "ds-idrepo-0" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "ds-idrepo-2" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "0" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "instance": "10.142.0.107:9100" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "instance": "10.142.0.43:9100" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "3924178420736" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "instance": "10.142.0.107:9100" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "instance": "10.142.0.43:9100" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "0.45210000000000244" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "ds-cts-0" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "ds-cts-2" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "0.2146666666666666" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "ds-idrepo-0" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "ds-idrepo-2" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "0" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644149" 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "ds-idrepo-0" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "ds-idrepo-2" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "0" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "1876.0666666666666" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "instance": "10.142.0.107:9100" 21:15:49 DEBUG "instance": "10.142.0.107:9100" 21:15:49 DEBUG }, 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "instance": "10.142.0.43:9100" 21:15:49 DEBUG "instance": "10.142.0.43:9100" 21:15:49 DEBUG }, 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "62454239.5" 21:15:49 DEBUG "62449839.599999994" 21:15:49 DEBUG ] 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "0" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "36.5715" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "373832.5" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "12.007200680454094" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644149" 21:15:49 INFO [http_cmd]: http status code OK 21:15:49 DEBUG --- status code --- 21:15:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:49 DEBUG --- http response --- 21:15:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:49 DEBUG ----- output ----- 21:15:49 DEBUG { 21:15:49 DEBUG "status": "success", 21:15:49 DEBUG "data": { 21:15:49 DEBUG "resultType": "vector", 21:15:49 DEBUG "result": [ 21:15:49 DEBUG { 21:15:49 DEBUG "metric": { 21:15:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG [...] 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG 21:15:49 DEBUG "pod": "ds-idrepo-0" 21:15:49 DEBUG }, 21:15:49 DEBUG "value": [ 21:15:49 DEBUG 1713644149, 21:15:49 DEBUG "5268.783254399813" 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG ] 21:15:49 DEBUG } 21:15:49 DEBUG } 21:15:50 INFO 21:15:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:15:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:15:50 INFO [loop_until]: OK (rc = 0) 21:15:50 DEBUG --- stdout --- 21:15:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 13892m 8820Mi am-767f6b67b5-qcqz2 13976m 8938Mi am-767f6b67b5-vbltz 15653m 9174Mi ds-cts-0 20429m 12212Mi ds-cts-1 18990m 12223Mi ds-cts-2 19708m 12366Mi ds-idrepo-0 5949m 3942Mi ds-idrepo-1 5856m 3934Mi ds-idrepo-2 6108m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 12771m 8463Mi 21:15:50 DEBUG --- stderr --- 21:15:50 DEBUG 21:15:50 INFO 21:15:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:15:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:15:50 INFO [loop_until]: OK (rc = 0) 21:15:50 DEBUG --- stdout --- 21:15:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6287Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5806m 19% 7703Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13814m 46% 11566Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19487m 65% 15430Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 13972m 46% 11877Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 11983m 40% 3824Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 15812m 52% 11421Mi 10% gke-xlou-cdm-ds-0f601751-j488 6069m 20% 7273Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18842m 63% 17235Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 5991m 20% 7924Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19703m 65% 15050Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11856m 74% 14559Mi 24% 21:15:50 DEBUG --- stderr --- 21:15:50 DEBUG 21:16:48 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644208" 21:16:48 INFO [http_cmd]: http status code OK 21:16:48 DEBUG --- status code --- 21:16:48 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:48 DEBUG --- http response --- 21:16:48 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:48 DEBUG ----- output ----- 21:16:48 DEBUG { 21:16:48 DEBUG "status": "success", 21:16:48 DEBUG "data": { 21:16:48 DEBUG "resultType": "vector", 21:16:48 DEBUG "result": [ 21:16:48 DEBUG { 21:16:48 DEBUG "metric": { 21:16:48 DEBUG "instance": "10.142.0.107:9100" 21:16:48 DEBUG }, 21:16:48 DEBUG "value": [ 21:16:48 DEBUG 21:16:48 DEBUG 21:16:48 DEBUG 21:16:48 DEBUG [...] 21:16:48 DEBUG 21:16:48 DEBUG 21:16:48 DEBUG 21:16:48 DEBUG "instance": "10.142.0.43:9100" 21:16:48 DEBUG }, 21:16:48 DEBUG "value": [ 21:16:48 DEBUG 1713644208, 21:16:48 DEBUG "0.08100000000000404" 21:16:48 DEBUG ] 21:16:48 DEBUG } 21:16:48 DEBUG ] 21:16:48 DEBUG } 21:16:48 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 ERROR [http_cmd]: ERROR 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 ERROR --- status code --- 21:16:49 DEBUG --- status code --- 21:16:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 ERROR --- http response --- 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:16:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "1380.6666666666667" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "ds-idrepo-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-cts-2" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "0" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 DEBUG "result": [ 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "pod": "ds-cts-0" 21:16:49 DEBUG "data": { 21:16:49 DEBUG }, 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "value": [ 21:16:49 DEBUG "result": [ 21:16:49 DEBUG 21:16:49 DEBUG { 21:16:49 DEBUG 21:16:49 DEBUG "metric": { 21:16:49 DEBUG 21:16:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:49 DEBUG [...] 21:16:49 DEBUG }, 21:16:49 DEBUG 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-cts-2" 21:16:49 DEBUG 21:16:49 DEBUG }, 21:16:49 DEBUG [...] 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG 21:16:49 DEBUG "0" 21:16:49 DEBUG 21:16:49 DEBUG ] 21:16:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:49 DEBUG } 21:16:49 DEBUG }, 21:16:49 DEBUG ] 21:16:49 DEBUG "value": [ 21:16:49 DEBUG } 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG } 21:16:49 DEBUG "1380.2" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG { 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:16:49 DEBUG "data": { 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG { 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG "status": "success", 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644209" 21:16:49 DEBUG "data": { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG "data": { 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG "result": [ 21:16:49 DEBUG }, 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG "result": [ 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG { 21:16:49 DEBUG "value": [ 21:16:49 DEBUG { 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG "metric": { 21:16:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:16:49 DEBUG 21:16:49 DEBUG "metric": { 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:16:49 DEBUG 21:16:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:49 DEBUG { 21:16:49 DEBUG }, 21:16:49 DEBUG 21:16:49 DEBUG }, 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "value": [ 21:16:49 DEBUG [...] 21:16:49 DEBUG "value": [ 21:16:49 DEBUG "data": { 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "result": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG { 21:16:49 DEBUG [...] 21:16:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:49 DEBUG [...] 21:16:49 DEBUG "metric": { 21:16:49 DEBUG 21:16:49 DEBUG }, 21:16:49 DEBUG 21:16:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:16:49 DEBUG 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG }, 21:16:49 DEBUG 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG 21:16:49 DEBUG "value": [ 21:16:49 DEBUG "pod": "ds-cts-0" 21:16:49 DEBUG "500" 21:16:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:49 DEBUG 21:16:49 DEBUG }, 21:16:49 DEBUG ] 21:16:49 DEBUG }, 21:16:49 DEBUG 21:16:49 DEBUG "value": [ 21:16:49 DEBUG } 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG ] 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG [...] 21:16:49 DEBUG "457689157.48453754" 21:16:49 DEBUG } 21:16:49 DEBUG "0" 21:16:49 DEBUG 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 DEBUG 21:16:49 DEBUG ] 21:16:49 DEBUG ] 21:16:49 DEBUG "pod": "ds-cts-0" 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 DEBUG }, 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "75676202.92445746" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "ds-cts-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-cts-2" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "29.966666666666665" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "13516754944" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "ds-idrepo-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-idrepo-2" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "0" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "instance": "10.142.0.107:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "instance": "10.142.0.43:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "3928378471424" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "instance": "10.142.0.107:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "instance": "10.142.0.43:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "0.46130000000003785" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "ds-idrepo-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-idrepo-2" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "0" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "ds-cts-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-cts-2" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "29.03543333333333" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "instance": "10.142.0.107:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "instance": "10.142.0.43:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "47998646.766666666" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "ds-idrepo-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-idrepo-2" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "0" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "1380.3" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "instance": "10.142.0.107:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "instance": "10.142.0.43:9100" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "47993672.5" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-cts-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "18.207143827536843" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "11.982571670461382" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "38.6155" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "364477.5" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644209" 21:16:49 INFO [http_cmd]: http status code OK 21:16:49 DEBUG --- status code --- 21:16:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:49 DEBUG --- http response --- 21:16:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:49 DEBUG ----- output ----- 21:16:49 DEBUG { 21:16:49 DEBUG "status": "success", 21:16:49 DEBUG "data": { 21:16:49 DEBUG "resultType": "vector", 21:16:49 DEBUG "result": [ 21:16:49 DEBUG { 21:16:49 DEBUG "metric": { 21:16:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG [...] 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG 21:16:49 DEBUG "pod": "ds-cts-0" 21:16:49 DEBUG }, 21:16:49 DEBUG "value": [ 21:16:49 DEBUG 1713644209, 21:16:49 DEBUG "73025247.00582746" 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG ] 21:16:49 DEBUG } 21:16:49 DEBUG } 21:16:50 INFO 21:16:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:16:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:16:50 INFO [loop_until]: OK (rc = 0) 21:16:50 DEBUG --- stdout --- 21:16:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 8130m 8819Mi am-767f6b67b5-qcqz2 6832m 8926Mi am-767f6b67b5-vbltz 8160m 9010Mi ds-cts-0 18419m 12210Mi ds-cts-1 17345m 12224Mi ds-cts-2 18166m 12364Mi ds-idrepo-0 3608m 3941Mi ds-idrepo-1 3352m 3948Mi ds-idrepo-2 3722m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11986m 12798Mi 21:16:50 DEBUG --- stderr --- 21:16:50 DEBUG 21:16:50 INFO 21:16:50 INFO [loop_until]: kubectl --namespace=xlou top node 21:16:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:16:51 INFO [loop_until]: OK (rc = 0) 21:16:51 DEBUG --- stdout --- 21:16:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6301Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3464m 11% 7703Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7853m 26% 11565Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3344Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 17714m 59% 15437Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5190Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8297m 27% 11846Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8777m 29% 3962Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 8670m 29% 11272Mi 10% gke-xlou-cdm-ds-0f601751-j488 3385m 11% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17544m 58% 17220Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 3620m 12% 7939Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18770m 62% 15067Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 12478m 78% 17539Mi 29% 21:16:51 DEBUG --- stderr --- 21:16:51 DEBUG 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "instance": "10.142.0.107:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "instance": "10.142.0.43:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "0.1033333333333303" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 ERROR [http_cmd]: ERROR 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 ERROR --- status code --- 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG --- status code --- 21:17:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:17:49 DEBUG { 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 ERROR --- http response --- 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG "data": { 21:17:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:17:49 DEBUG "result": [ 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "pod": "ds-cts-0" 21:17:49 DEBUG "data": { 21:17:49 DEBUG }, 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "result": [ 21:17:49 DEBUG 21:17:49 DEBUG { 21:17:49 DEBUG 21:17:49 DEBUG "metric": { 21:17:49 DEBUG 21:17:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:49 DEBUG [...] 21:17:49 DEBUG }, 21:17:49 DEBUG 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-cts-2" 21:17:49 DEBUG 21:17:49 DEBUG }, 21:17:49 DEBUG [...] 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG 21:17:49 DEBUG "0" 21:17:49 DEBUG 21:17:49 DEBUG ] 21:17:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:49 DEBUG } 21:17:49 DEBUG }, 21:17:49 DEBUG ] 21:17:49 DEBUG "value": [ 21:17:49 DEBUG } 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG } 21:17:49 DEBUG "1397.1333333333334" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG { 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "data": { 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG "result": [ 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG { 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:49 DEBUG "pod": "ds-cts-0" 21:17:49 DEBUG "pod": "ds-idrepo-0" 21:17:49 DEBUG }, 21:17:49 DEBUG }, 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG [...] 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:49 DEBUG "pod": "ds-cts-2" 21:17:49 DEBUG "pod": "ds-cts-2" 21:17:49 DEBUG }, 21:17:49 DEBUG }, 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "0" 21:17:49 DEBUG "32" 21:17:49 DEBUG "0" 21:17:49 DEBUG ] 21:17:49 DEBUG ] 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG ] 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG { 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG "data": { 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG "result": [ 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:17:49 DEBUG { 21:17:49 DEBUG "data": { 21:17:49 DEBUG }, 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "data": { 21:17:49 DEBUG "result": [ 21:17:49 DEBUG 21:17:49 DEBUG "resultType": "vector", 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG { 21:17:49 DEBUG 21:17:49 DEBUG "result": [ 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG "metric": { 21:17:49 DEBUG 21:17:49 DEBUG { 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:17:49 DEBUG [...] 21:17:49 DEBUG "metric": { 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG }, 21:17:49 DEBUG 21:17:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:17:49 DEBUG "value": [ 21:17:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:17:49 DEBUG 21:17:49 DEBUG }, 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-cts-0" 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG }, 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG "371972776.84124863" 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG ] 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-cts-0" 21:17:49 DEBUG } 21:17:49 DEBUG 21:17:49 DEBUG }, 21:17:49 DEBUG ] 21:17:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:17:49 DEBUG "value": [ 21:17:49 DEBUG } 21:17:49 DEBUG }, 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG } 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "93530730.22147301" 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG ] 21:17:49 DEBUG "16724484096" 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "1396.2666666666667" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "4215.099999999999" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "ds-idrepo-0" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-idrepo-2" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "0" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "instance": "10.142.0.107:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "instance": "10.142.0.43:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "3932475954176" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "instance": "10.142.0.107:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "instance": "10.142.0.43:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "0.47146666666674164" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "ds-cts-0" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-cts-2" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "0.11563333333333503" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "ds-idrepo-0" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-idrepo-2" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "0" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "instance": "10.142.0.107:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "instance": "10.142.0.43:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "63727177.13333333" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "11.975877989084179" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "ds-idrepo-0" 21:17:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:49 DEBUG }, 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-idrepo-2" 21:17:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:49 DEBUG }, 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "0" 21:17:49 DEBUG "1396.1" 21:17:49 DEBUG ] 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "instance": "10.142.0.107:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "instance": "10.142.0.43:9100" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "63721740.3" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "41.283" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-cts-0" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "19.438650787650516" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "367533" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644269" 21:17:49 INFO [http_cmd]: http status code OK 21:17:49 DEBUG --- status code --- 21:17:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:49 DEBUG --- http response --- 21:17:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:17:49 DEBUG ----- output ----- 21:17:49 DEBUG { 21:17:49 DEBUG "status": "success", 21:17:49 DEBUG "data": { 21:17:49 DEBUG "resultType": "vector", 21:17:49 DEBUG "result": [ 21:17:49 DEBUG { 21:17:49 DEBUG "metric": { 21:17:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG [...] 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG 21:17:49 DEBUG "pod": "ds-cts-0" 21:17:49 DEBUG }, 21:17:49 DEBUG "value": [ 21:17:49 DEBUG 1713644269, 21:17:49 DEBUG "64252651.74678836" 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG ] 21:17:49 DEBUG } 21:17:49 DEBUG } 21:17:50 INFO 21:17:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:17:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:17:50 INFO [loop_until]: OK (rc = 0) 21:17:50 DEBUG --- stdout --- 21:17:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 11233m 8814Mi am-767f6b67b5-qcqz2 11513m 8926Mi am-767f6b67b5-vbltz 11733m 9008Mi ds-cts-0 18931m 12200Mi ds-cts-1 18421m 12222Mi ds-cts-2 20263m 12364Mi ds-idrepo-0 4128m 3942Mi ds-idrepo-1 4705m 3950Mi ds-idrepo-2 4766m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 12119m 15791Mi 21:17:50 DEBUG --- stderr --- 21:17:50 DEBUG 21:17:51 INFO 21:17:51 INFO [loop_until]: kubectl --namespace=xlou top node 21:17:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:17:51 INFO [loop_until]: OK (rc = 0) 21:17:51 DEBUG --- stdout --- 21:17:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4680m 15% 7708Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11953m 40% 11560Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3334Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19205m 64% 15423Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5188Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11669m 39% 11842Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8025m 26% 3892Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 13014m 43% 11264Mi 10% gke-xlou-cdm-ds-0f601751-j488 5125m 17% 7277Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18320m 61% 17204Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 4993m 16% 7938Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19994m 66% 15060Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 12509m 78% 19903Mi 33% 21:17:51 DEBUG --- stderr --- 21:17:51 DEBUG 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "1730.9" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 ERROR [http_cmd]: ERROR 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 ERROR --- status code --- 21:18:49 DEBUG --- status code --- 21:18:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 ERROR --- http response --- 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:18:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "ds-cts-0" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "ds-cts-2" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "29.333333333333332" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:18:49 DEBUG "pod": "ds-idrepo-0" 21:18:49 DEBUG }, 21:18:49 DEBUG "instance": "10.142.0.107:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:18:49 DEBUG 21:18:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:18:49 DEBUG "pod": "ds-cts-2" 21:18:49 DEBUG }, 21:18:49 DEBUG "instance": "10.142.0.43:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "1731.3333333333333" 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0" 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG "0.07433333333333772" 21:18:49 DEBUG ] 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "ds-cts-0" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "ds-cts-2" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG --- status code --- 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG { 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG "data": { 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG "result": [ 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG { 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG }, 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG "value": [ 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:18:49 DEBUG 21:18:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:18:49 DEBUG }, 21:18:49 DEBUG [...] 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG }, 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "2708.833333333333" 21:18:49 DEBUG 21:18:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:18:49 DEBUG ] 21:18:49 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:18:49 DEBUG }, 21:18:49 DEBUG } 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG ] 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG } 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "16878313472" 21:18:49 DEBUG } 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "ds-idrepo-0" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "ds-idrepo-2" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "instance": "10.142.0.107:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "instance": "10.142.0.43:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "3936725677056" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "instance": "10.142.0.107:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "instance": "10.142.0.43:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0.4210666666666735" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "ds-cts-0" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "ds-cts-2" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "28.89283333333333" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "ds-idrepo-0" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "ds-idrepo-2" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "instance": "10.142.0.107:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "instance": "10.142.0.43:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "55152533.099999994" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "44.1565" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "instance": "10.142.0.107:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "instance": "10.142.0.43:9100" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "55147434.266666666" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "11.999697533761823" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG { 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:18:49 DEBUG }, 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG "value": [ 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG 21:18:49 DEBUG { 21:18:49 DEBUG 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG 21:18:49 DEBUG "data": { 21:18:49 DEBUG [...] 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG 21:18:49 DEBUG "result": [ 21:18:49 DEBUG 21:18:49 DEBUG { 21:18:49 DEBUG 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:18:49 DEBUG "pod": "ds-idrepo-0" 21:18:49 DEBUG }, 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG 21:18:49 DEBUG "1732.6666666666667" 21:18:49 DEBUG 21:18:49 DEBUG ] 21:18:49 DEBUG 21:18:49 DEBUG } 21:18:49 DEBUG [...] 21:18:49 DEBUG ] 21:18:49 DEBUG 21:18:49 DEBUG } 21:18:49 DEBUG 21:18:49 DEBUG } 21:18:49 DEBUG 21:18:49 DEBUG "pod": "ds-idrepo-2" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "0" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "301944" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644329" 21:18:49 INFO [http_cmd]: http status code OK 21:18:49 DEBUG --- status code --- 21:18:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:18:49 DEBUG --- http response --- 21:18:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:18:49 DEBUG ----- output ----- 21:18:49 DEBUG { 21:18:49 DEBUG "status": "success", 21:18:49 DEBUG "data": { 21:18:49 DEBUG "resultType": "vector", 21:18:49 DEBUG "result": [ 21:18:49 DEBUG { 21:18:49 DEBUG "metric": { 21:18:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG [...] 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG 21:18:49 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:18:49 DEBUG }, 21:18:49 DEBUG "value": [ 21:18:49 DEBUG 1713644329, 21:18:49 DEBUG "1041.6369227917885" 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG ] 21:18:49 DEBUG } 21:18:49 DEBUG } 21:18:50 INFO 21:18:50 INFO [loop_until]: kubectl --namespace=xlou top pods 21:18:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:18:51 INFO [loop_until]: OK (rc = 0) 21:18:51 DEBUG --- stdout --- 21:18:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 11385m 8816Mi am-767f6b67b5-qcqz2 11543m 8925Mi am-767f6b67b5-vbltz 12971m 9014Mi ds-cts-0 19105m 12207Mi ds-cts-1 18768m 12229Mi ds-cts-2 19275m 12363Mi ds-idrepo-0 4975m 3932Mi ds-idrepo-1 4969m 3949Mi ds-idrepo-2 5196m 3872Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 11613m 16002Mi 21:18:51 DEBUG --- stderr --- 21:18:51 DEBUG 21:18:51 INFO 21:18:51 INFO [loop_until]: kubectl --namespace=xlou top node 21:18:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:18:51 INFO [loop_until]: OK (rc = 0) 21:18:51 DEBUG --- stdout --- 21:18:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4937m 16% 7697Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11633m 38% 11564Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3343Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19063m 63% 15394Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5213Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12005m 40% 11841Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 10928m 36% 3821Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 12818m 42% 11273Mi 10% gke-xlou-cdm-ds-0f601751-j488 5261m 17% 7276Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18631m 62% 17233Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 5198m 17% 7926Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19347m 64% 15045Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 12657m 79% 19609Mi 33% 21:18:51 DEBUG --- stderr --- 21:18:51 DEBUG 21:19:41 WARNING File /tmp/lodemon_data-20240420_204036/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [20/Apr/2024 21:19:41] "GET /monitoring/average?start_time=20240420_201504&stop_time=20240420_201740 HTTP/1.1" 200 - 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644389" 21:19:49 ERROR [http_cmd]: ERROR 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 ERROR --- status code --- 21:19:49 DEBUG --- status code --- 21:19:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 ERROR --- http response --- 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:19:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "629.7666666666667" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:19:49 DEBUG --- status code --- 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG { 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG "data": { 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "result": [ 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG { 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "pod": "ds-cts-0" 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG }, 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "value": [ 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "ds-cts-0" 21:19:49 DEBUG 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "ds-idrepo-0" 21:19:49 DEBUG }, 21:19:49 DEBUG 21:19:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG [...] 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG "pod": "ds-cts-2" 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG }, 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "ds-cts-2" 21:19:49 DEBUG "0" 21:19:49 DEBUG 21:19:49 DEBUG "pod": "ds-cts-2" 21:19:49 DEBUG }, 21:19:49 DEBUG ] 21:19:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG } 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG ] 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "29.666666666666668" 21:19:49 DEBUG } 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "0" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG "113930240" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "instance": "10.142.0.107:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "instance": "10.142.0.43:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "0.10366666666668418" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "3991.09131403118" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:19:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:19:49 DEBUG }, 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:19:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:19:49 DEBUG }, 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "0" 21:19:49 DEBUG "632.3666666666667" 21:19:49 DEBUG ] 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "2380.5333333333333" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "ds-cts-0" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "52821917.50272231" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "ds-idrepo-0" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "ds-idrepo-2" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "0" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "instance": "10.142.0.107:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "instance": "10.142.0.43:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "3940377660416" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "instance": "10.142.0.107:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "instance": "10.142.0.43:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "0.5129666666666803" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "ds-cts-0" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "ds-cts-2" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "27.973533333333332" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "ds-idrepo-0" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "ds-idrepo-2" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "0" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "46.923500000000004" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "instance": "10.142.0.107:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "instance": "10.142.0.43:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "15785558.133333333" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "instance": "10.142.0.107:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "instance": "10.142.0.43:9100" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "15778556.4" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "9.55801549192396" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "0" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:19:49 DEBUG "pod": "ds-idrepo-0" 21:19:49 DEBUG }, 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:19:49 DEBUG "pod": "ds-idrepo-2" 21:19:49 DEBUG }, 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "632.5666666666666" 21:19:49 DEBUG "0" 21:19:49 DEBUG ] 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "289322.5" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644389" 21:19:49 INFO [http_cmd]: http status code OK 21:19:49 DEBUG --- status code --- 21:19:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:19:49 DEBUG --- http response --- 21:19:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:19:49 DEBUG ----- output ----- 21:19:49 DEBUG { 21:19:49 DEBUG "status": "success", 21:19:49 DEBUG "data": { 21:19:49 DEBUG "resultType": "vector", 21:19:49 DEBUG "result": [ 21:19:49 DEBUG { 21:19:49 DEBUG "metric": { 21:19:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG [...] 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG 21:19:49 DEBUG "pod": "ds-cts-0" 21:19:49 DEBUG }, 21:19:49 DEBUG "value": [ 21:19:49 DEBUG 1713644389, 21:19:49 DEBUG "26476602.965620942" 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG ] 21:19:49 DEBUG } 21:19:49 DEBUG } 21:19:51 INFO 21:19:51 INFO [loop_until]: kubectl --namespace=xlou top pods 21:19:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:19:51 INFO [loop_until]: OK (rc = 0) 21:19:51 DEBUG --- stdout --- 21:19:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 5535m 8815Mi am-767f6b67b5-qcqz2 7238m 8922Mi am-767f6b67b5-vbltz 4871m 9000Mi ds-cts-0 19672m 12200Mi ds-cts-1 17533m 12221Mi ds-cts-2 19441m 12366Mi ds-idrepo-0 2727m 3933Mi ds-idrepo-1 2400m 3949Mi ds-idrepo-2 2025m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 7m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 9334m 16Mi 21:19:51 DEBUG --- stderr --- 21:19:51 DEBUG 21:19:51 INFO 21:19:51 INFO [loop_until]: kubectl --namespace=xlou top node 21:19:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:19:51 INFO [loop_until]: OK (rc = 0) 21:19:51 DEBUG --- stdout --- 21:19:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6278Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2006m 6% 7702Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4792m 16% 11593Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3347Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 19134m 64% 15414Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5195Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6416m 21% 11862Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 5270m 17% 3676Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 5907m 19% 11290Mi 10% gke-xlou-cdm-ds-0f601751-j488 1496m 5% 7278Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17309m 57% 17217Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 1807m 6% 7927Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19236m 64% 15061Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 9056m 56% 5749Mi 9% 21:19:51 DEBUG --- stderr --- 21:19:51 DEBUG 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "ds-cts-0" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "ds-cts-2" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "113487872" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "ds-cts-0" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "ds-cts-2" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "29.666666666666668" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "instance": "10.142.0.107:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "instance": "10.142.0.43:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.1400000000000072" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 ERROR [http_cmd]: ERROR 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 ERROR --- status code --- 21:20:49 DEBUG --- status code --- 21:20:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 ERROR --- http response --- 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:20:49 DEBUG "pod": "ds-idrepo-0" 21:20:49 DEBUG }, 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG "pod": "ds-cts-2" 21:20:49 DEBUG }, 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.06666666666666667" 21:20:49 DEBUG "0" 21:20:49 DEBUG ] 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.06666666666666667" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "1896" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "25659451.224532392" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "ds-idrepo-0" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "ds-idrepo-2" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.06666666666666667" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "instance": "10.142.0.107:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "instance": "10.142.0.43:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "3942830578688" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "instance": "10.142.0.107:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "instance": "10.142.0.43:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.24009999999994325" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "ds-cts-0" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "ds-cts-2" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "24.444200000000002" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "ds-idrepo-0" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "ds-idrepo-2" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.06666666666666667" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "48.86" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "instance": "10.142.0.107:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "instance": "10.142.0.43:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "11897962.533333331" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "instance": "10.142.0.107:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "instance": "10.142.0.43:9100" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "11891045.6" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "ds-idrepo-0" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "ds-idrepo-2" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.06666666666666667" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644449" 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.024642773475161868" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "0.13333333333333333" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "201692" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644449" 21:20:49 INFO [http_cmd]: http status code OK 21:20:49 DEBUG --- status code --- 21:20:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:20:49 DEBUG --- http response --- 21:20:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:20:49 DEBUG ----- output ----- 21:20:49 DEBUG { 21:20:49 DEBUG "status": "success", 21:20:49 DEBUG "data": { 21:20:49 DEBUG "resultType": "vector", 21:20:49 DEBUG "result": [ 21:20:49 DEBUG { 21:20:49 DEBUG "metric": { 21:20:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG [...] 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG 21:20:49 DEBUG "pod": "overseer-0-66cd7d7695-bdwll" 21:20:49 DEBUG }, 21:20:49 DEBUG "value": [ 21:20:49 DEBUG 1713644449, 21:20:49 DEBUG "3597.008145837749" 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG ] 21:20:49 DEBUG } 21:20:49 DEBUG } 21:20:51 INFO 21:20:51 INFO [loop_until]: kubectl --namespace=xlou top pods 21:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:20:51 INFO [loop_until]: OK (rc = 0) 21:20:51 DEBUG --- stdout --- 21:20:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2103m 8722Mi am-767f6b67b5-qcqz2 2149m 8829Mi am-767f6b67b5-vbltz 2252m 8883Mi ds-cts-0 18200m 12199Mi ds-cts-1 13403m 12205Mi ds-cts-2 18765m 12452Mi ds-idrepo-0 631m 3971Mi ds-idrepo-1 691m 4075Mi ds-idrepo-2 402m 3871Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-597977695c-ld9s4 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 1m 16Mi 21:20:51 DEBUG --- stderr --- 21:20:51 DEBUG 21:20:51 INFO 21:20:51 INFO [loop_until]: kubectl --namespace=xlou top node 21:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:20:51 INFO [loop_until]: OK (rc = 0) 21:20:51 DEBUG --- stdout --- 21:20:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 732m 2% 7697Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2122m 7% 11483Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 120m 0% 3378Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17843m 59% 15432Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2207m 7% 11748Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 126m 0% 3607Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 2316m 7% 11150Mi 9% gke-xlou-cdm-ds-0f601751-j488 762m 2% 7230Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 13130m 43% 17233Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 780m 2% 7903Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18911m 63% 15055Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 132m 0% 5654Mi 9% 21:20:51 DEBUG --- stderr --- 21:20:51 DEBUG 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "ds-cts-0" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "28.833333333333332" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 ERROR [http_cmd]: ERROR 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 ERROR --- status code --- 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG --- status code --- 21:21:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 ERROR --- http response --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:21:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:21:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "ds-cts-0" 21:21:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:21:49 DEBUG }, 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:21:49 DEBUG }, 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0" 21:21:49 DEBUG "1081057280" 21:21:49 DEBUG ] 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "instance": "10.142.0.107:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "instance": "10.142.0.43:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0.011333333333341975" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG { 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG "data": { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG { 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:21:49 DEBUG }, 21:21:49 DEBUG "data": { 21:21:49 DEBUG "pod": "ds-idrepo-0" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG "result": [ 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG { 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "metric": { 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG }, 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:21:49 DEBUG }, 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG [...] 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0" 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "1898.2" 21:21:49 DEBUG ] 21:21:49 DEBUG 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:21:49 DEBUG "0" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG }, 21:21:49 DEBUG { 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG "value": [ 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "data": { 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 DEBUG "461885583.8294535" 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG "result": [ 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG { 21:21:49 DEBUG } 21:21:49 DEBUG "metric": { 21:21:49 DEBUG } 21:21:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "105525844.7827172" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "ds-idrepo-0" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-idrepo-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "instance": "10.142.0.107:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "instance": "10.142.0.43:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "3944129285120" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "instance": "10.142.0.107:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "instance": "10.142.0.43:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0.016166666666570486" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "ds-cts-0" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "19.2915" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "instance": "10.142.0.107:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "instance": "10.142.0.43:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "11721527.933333334" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "instance": "10.142.0.107:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "instance": "10.142.0.43:9100" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "11728108.266666666" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "ds-idrepo-0" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-idrepo-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "49.260000000000005" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "ds-idrepo-0" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-idrepo-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "0" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644509" 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG ----- output ----- 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG { 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG "data": { 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG "result": [ 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:21:49 DEBUG { 21:21:49 DEBUG { 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "metric": { 21:21:49 DEBUG { 21:21:49 DEBUG "data": { 21:21:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG }, 21:21:49 DEBUG "data": { 21:21:49 DEBUG "result": [ 21:21:49 DEBUG "value": [ 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG { 21:21:49 DEBUG 21:21:49 DEBUG "result": [ 21:21:49 DEBUG "metric": { 21:21:49 DEBUG 21:21:49 DEBUG { 21:21:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:21:49 DEBUG 21:21:49 DEBUG "metric": { 21:21:49 DEBUG }, 21:21:49 DEBUG [...] 21:21:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG }, 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG }, 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "value": [ 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "0" 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG ] 21:21:49 DEBUG 21:21:49 DEBUG }, 21:21:49 DEBUG } 21:21:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:21:49 DEBUG "value": [ 21:21:49 DEBUG ] 21:21:49 DEBUG }, 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG } 21:21:49 DEBUG "value": [ 21:21:49 DEBUG "19.10705796584794" 21:21:49 DEBUG } 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG ] 21:21:49 DEBUG "1.033607492914459" 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "98702.5" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644509" 21:21:49 INFO [http_cmd]: http status code OK 21:21:49 DEBUG --- status code --- 21:21:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:21:49 DEBUG --- http response --- 21:21:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:21:49 DEBUG ----- output ----- 21:21:49 DEBUG { 21:21:49 DEBUG "status": "success", 21:21:49 DEBUG "data": { 21:21:49 DEBUG "resultType": "vector", 21:21:49 DEBUG "result": [ 21:21:49 DEBUG { 21:21:49 DEBUG "metric": { 21:21:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG [...] 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG 21:21:49 DEBUG "pod": "ds-cts-2" 21:21:49 DEBUG }, 21:21:49 DEBUG "value": [ 21:21:49 DEBUG 1713644509, 21:21:49 DEBUG "56916892.232006" 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG ] 21:21:49 DEBUG } 21:21:49 DEBUG } 21:21:51 INFO 21:21:51 INFO [loop_until]: kubectl --namespace=xlou top pods 21:21:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:21:51 INFO [loop_until]: OK (rc = 0) 21:21:51 DEBUG --- stdout --- 21:21:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2353m 8716Mi am-767f6b67b5-qcqz2 2140m 8826Mi am-767f6b67b5-vbltz 2149m 8882Mi ds-cts-0 18014m 12204Mi ds-cts-1 4874m 12218Mi ds-cts-2 20004m 12350Mi ds-idrepo-0 1175m 3993Mi ds-idrepo-1 795m 3977Mi ds-idrepo-2 972m 4009Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-597977695c-ld9s4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 1008m 939Mi 21:21:51 DEBUG --- stderr --- 21:21:51 DEBUG 21:21:51 INFO 21:21:51 INFO [loop_until]: kubectl --namespace=xlou top node 21:21:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:21:51 INFO [loop_until]: OK (rc = 0) 21:21:51 DEBUG --- stdout --- 21:21:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1170m 3% 7720Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2230m 7% 11471Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3373Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18311m 61% 15429Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5210Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2279m 7% 11755Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 3526Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 2279m 7% 11154Mi 9% gke-xlou-cdm-ds-0f601751-j488 883m 2% 7268Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 4543m 15% 17250Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 895m 2% 7925Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18915m 63% 15034Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 6391Mi 10% 21:21:51 DEBUG --- stderr --- 21:21:51 DEBUG 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "ds-cts-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-cts-2" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG { 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "data": { 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "result": [ 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG }, 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:22:49 DEBUG "value": [ 21:22:49 DEBUG "pod": "ds-cts-0" 21:22:49 DEBUG }, 21:22:49 DEBUG 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 ERROR [http_cmd]: ERROR 21:22:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 ERROR --- status code --- 21:22:49 DEBUG }, 21:22:49 DEBUG 21:22:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:22:49 DEBUG --- status code --- 21:22:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:22:49 DEBUG "value": [ 21:22:49 DEBUG "pod": "ds-cts-2" 21:22:49 DEBUG }, 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 ERROR --- http response --- 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG "0" 21:22:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG ] 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG } 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG "29.633333333333333" 21:22:49 DEBUG ] 21:22:49 DEBUG ] 21:22:49 DEBUG { 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG "data": { 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 DEBUG "result": [ 21:22:49 DEBUG } 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "2342207488" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "ds-idrepo-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-cts-2" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "instance": "10.142.0.107:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "instance": "10.142.0.43:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0.0006666666666660604" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-cts-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "131838569.45408857" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "636.1666666666666" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-cts-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "423035822.63664323" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "ds-idrepo-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-idrepo-2" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "instance": "10.142.0.107:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "instance": "10.142.0.43:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "3944294519808" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "instance": "10.142.0.107:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "instance": "10.142.0.43:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0.0017999999999621726" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "ds-cts-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-cts-2" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "13.974866666666667" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "instance": "10.142.0.107:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "instance": "10.142.0.43:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "33178.03333333333" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "instance": "10.142.0.107:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "instance": "10.142.0.43:9100" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "25984.166666666664" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "ds-idrepo-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-idrepo-2" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-cts-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "17.059506034451818" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "49.377" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "ds-idrepo-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-idrepo-2" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "0" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "1.0353470480124694" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "9694" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644569" 21:22:49 INFO [http_cmd]: http status code OK 21:22:49 DEBUG --- status code --- 21:22:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:22:49 DEBUG --- http response --- 21:22:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:22:49 DEBUG ----- output ----- 21:22:49 DEBUG { 21:22:49 DEBUG "status": "success", 21:22:49 DEBUG "data": { 21:22:49 DEBUG "resultType": "vector", 21:22:49 DEBUG "result": [ 21:22:49 DEBUG { 21:22:49 DEBUG "metric": { 21:22:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG [...] 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG 21:22:49 DEBUG "pod": "ds-cts-0" 21:22:49 DEBUG }, 21:22:49 DEBUG "value": [ 21:22:49 DEBUG 1713644569, 21:22:49 DEBUG "81150721.8199253" 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG ] 21:22:49 DEBUG } 21:22:49 DEBUG } 21:22:51 INFO 21:22:51 INFO [loop_until]: kubectl --namespace=xlou top pods 21:22:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:22:51 INFO [loop_until]: OK (rc = 0) 21:22:51 DEBUG --- stdout --- 21:22:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1348m 8715Mi am-767f6b67b5-qcqz2 944m 8827Mi am-767f6b67b5-vbltz 758m 8871Mi ds-cts-0 16678m 12203Mi ds-cts-1 1801m 12217Mi ds-cts-2 18047m 12294Mi ds-idrepo-0 42m 3983Mi ds-idrepo-1 242m 3977Mi ds-idrepo-2 147m 3885Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-597977695c-ld9s4 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-66cd7d7695-bdwll 987m 1865Mi 21:22:51 DEBUG --- stderr --- 21:22:51 DEBUG 21:22:51 INFO 21:22:51 INFO [loop_until]: kubectl --namespace=xlou top node 21:22:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:22:51 INFO [loop_until]: OK (rc = 0) 21:22:51 DEBUG --- stdout --- 21:22:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 145m 0% 7739Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 922m 3% 11451Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 16945m 56% 15414Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1177m 3% 11740Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 3473Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 972m 3% 11139Mi 9% gke-xlou-cdm-ds-0f601751-j488 104m 0% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 1770m 5% 17235Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 141m 0% 7948Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18242m 61% 14968Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1045m 6% 7525Mi 12% 21:22:51 DEBUG --- stderr --- 21:22:51 DEBUG 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG "metric": { 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:23:49 DEBUG { 21:23:49 DEBUG }, 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "value": [ 21:23:49 DEBUG "data": { 21:23:49 DEBUG 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG 21:23:49 DEBUG "result": [ 21:23:49 DEBUG 21:23:49 DEBUG { 21:23:49 DEBUG [...] 21:23:49 DEBUG "metric": { 21:23:49 DEBUG 21:23:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:23:49 DEBUG 21:23:49 DEBUG }, 21:23:49 DEBUG 21:23:49 DEBUG "value": [ 21:23:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:23:49 DEBUG 21:23:49 DEBUG }, 21:23:49 DEBUG 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG [...] 21:23:49 DEBUG "0" 21:23:49 DEBUG 21:23:49 DEBUG ] 21:23:49 DEBUG 21:23:49 DEBUG } 21:23:49 DEBUG 21:23:49 DEBUG ] 21:23:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:23:49 DEBUG } 21:23:49 DEBUG }, 21:23:49 DEBUG } 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644629" 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "2749214720" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "ds-cts-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-cts-2" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "instance": "10.142.0.107:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "instance": "10.142.0.43:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 ERROR [http_cmd]: ERROR 21:23:49 ERROR --- status code --- 21:23:49 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:23:49 ERROR --- http response --- 21:23:49 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:23:49 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG { 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "ds-cts-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-cts-2" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "30" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "23573.234780823324" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-cts-2" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-idrepo-2" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0.06666666666666667" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "instance": "10.142.0.107:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "instance": "10.142.0.43:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "3944302361600" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "instance": "10.142.0.107:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "instance": "10.142.0.43:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0.0009666666666807563" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "ds-cts-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-cts-2" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "instance": "10.142.0.107:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "instance": "10.142.0.43:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "34708.833333333336" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "instance": "10.142.0.107:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "instance": "10.142.0.43:9100" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "27605.333333333332" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-idrepo-2" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0.009067386510763204" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "49.377" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG "data": { 21:23:49 DEBUG { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG "data": { 21:23:49 DEBUG { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "result": [ 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG { 21:23:49 DEBUG }, 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "value": [ 21:23:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:23:49 DEBUG 21:23:49 DEBUG }, 21:23:49 DEBUG 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-idrepo-2" 21:23:49 DEBUG 21:23:49 DEBUG }, 21:23:49 DEBUG 21:23:49 DEBUG "value": [ 21:23:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG }, 21:23:49 DEBUG "0" 21:23:49 DEBUG "value": [ 21:23:49 DEBUG ] 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG } 21:23:49 DEBUG "0" 21:23:49 DEBUG ] 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "0.7500582991178832" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "am-767f6b67b5-9zw22" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "am-767f6b67b5-vbltz" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "1" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713644629" 21:23:49 INFO [http_cmd]: http status code OK 21:23:49 DEBUG --- status code --- 21:23:49 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:23:49 DEBUG --- http response --- 21:23:49 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:23:49 DEBUG ----- output ----- 21:23:49 DEBUG { 21:23:49 DEBUG "status": "success", 21:23:49 DEBUG "data": { 21:23:49 DEBUG "resultType": "vector", 21:23:49 DEBUG "result": [ 21:23:49 DEBUG { 21:23:49 DEBUG "metric": { 21:23:49 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG [...] 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG 21:23:49 DEBUG "pod": "ds-idrepo-0" 21:23:49 DEBUG }, 21:23:49 DEBUG "value": [ 21:23:49 DEBUG 1713644629, 21:23:49 DEBUG "2521.787906785751" 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG ] 21:23:49 DEBUG } 21:23:49 DEBUG } 21:23:51 INFO Finished: True 21:23:51 INFO Waiting for threads to register finish flag 21:24:49 INFO Done. Have a nice day! :) 127.0.0.1 - - [20/Apr/2024 21:24:49] "GET /monitoring/stop HTTP/1.1" 200 - 21:24:52 WARNING File /tmp/lodemon_data-20240420_204036/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6481', ...) 127.0.0.1 - - [20/Apr/2024 21:24:56] "GET /monitoring/process HTTP/1.1" 200 -