==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-5cd77b55c5-wc625 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Sun, 21 Apr 2024 19:35:23 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=5cd77b55c5 skaffold.dev/run-id=6ef1ce61-ee0b-4983-961d-2896745be4af Annotations: Status: Running IP: 10.106.42.194 IPs: IP: 10.106.42.194 Controlled By: ReplicaSet/lodemon-5cd77b55c5 Containers: lodemon: Container ID: containerd://e3bc44652d083d026fbb8033ef65b224f745b1bee7c62cf0effbb8deee154d2f Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 21 Apr 2024 19:35:24 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-69f95 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-69f95: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43m default-scheduler Successfully assigned xlou/lodemon-5cd77b55c5-wc625 to gke-xlou-cdm-default-pool-1b0769c8-7vmj Normal Pulling 43m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 43m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 269.381162ms (269.387431ms including waiting) Normal Created 43m kubelet Created container lodemon Normal Started 43m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.42.194:8080 Press CTRL+C to quit 20:36:30 INFO 20:36:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:30 INFO [loop_until]: OK (rc = 0) 20:36:30 DEBUG --- stdout --- 20:36:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:30 DEBUG --- stderr --- 20:36:30 DEBUG 20:36:31 INFO 20:36:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:31 INFO [loop_until]: OK (rc = 0) 20:36:31 DEBUG --- stdout --- 20:36:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:31 DEBUG --- stderr --- 20:36:31 DEBUG 20:36:31 INFO 20:36:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:31 INFO [loop_until]: OK (rc = 0) 20:36:31 DEBUG --- stdout --- 20:36:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:31 DEBUG --- stderr --- 20:36:31 DEBUG 20:36:31 INFO 20:36:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:31 INFO [loop_until]: OK (rc = 0) 20:36:31 DEBUG --- stdout --- 20:36:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:31 DEBUG --- stderr --- 20:36:31 DEBUG 20:36:31 INFO 20:36:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:31 INFO [loop_until]: OK (rc = 0) 20:36:31 DEBUG --- stdout --- 20:36:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:31 DEBUG --- stderr --- 20:36:31 DEBUG 20:36:31 INFO 20:36:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:31 INFO [loop_until]: OK (rc = 0) 20:36:31 DEBUG --- stdout --- 20:36:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:31 DEBUG --- stderr --- 20:36:31 DEBUG 20:36:31 INFO 20:36:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:31 INFO [loop_until]: OK (rc = 0) 20:36:31 DEBUG --- stdout --- 20:36:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:31 DEBUG --- stderr --- 20:36:31 DEBUG 20:36:31 INFO 20:36:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:31 INFO [loop_until]: OK (rc = 0) 20:36:31 DEBUG --- stdout --- 20:36:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:31 DEBUG --- stderr --- 20:36:31 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:32 INFO [loop_until]: OK (rc = 0) 20:36:32 DEBUG --- stdout --- 20:36:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:32 DEBUG --- stderr --- 20:36:32 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:32 INFO [loop_until]: OK (rc = 0) 20:36:32 DEBUG --- stdout --- 20:36:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:32 DEBUG --- stderr --- 20:36:32 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:32 INFO [loop_until]: OK (rc = 0) 20:36:32 DEBUG --- stdout --- 20:36:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:32 DEBUG --- stderr --- 20:36:32 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:32 INFO [loop_until]: OK (rc = 0) 20:36:32 DEBUG --- stdout --- 20:36:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:32 DEBUG --- stderr --- 20:36:32 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:32 INFO [loop_until]: OK (rc = 0) 20:36:32 DEBUG --- stdout --- 20:36:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:32 DEBUG --- stderr --- 20:36:32 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:32 INFO [loop_until]: OK (rc = 0) 20:36:32 DEBUG --- stdout --- 20:36:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:32 DEBUG --- stderr --- 20:36:32 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:32 INFO [loop_until]: OK (rc = 0) 20:36:32 DEBUG --- stdout --- 20:36:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:32 DEBUG --- stderr --- 20:36:32 DEBUG 20:36:32 INFO 20:36:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:33 INFO 20:36:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:33 INFO 20:36:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:33 INFO 20:36:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:33 INFO 20:36:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:33 INFO 20:36:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:33 INFO 20:36:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:33 INFO 20:36:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:33 INFO [loop_until]: OK (rc = 0) 20:36:33 DEBUG --- stdout --- 20:36:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:33 DEBUG --- stderr --- 20:36:33 DEBUG 20:36:34 INFO 20:36:34 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:34 INFO [loop_until]: OK (rc = 0) 20:36:34 DEBUG --- stdout --- 20:36:34 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:34 DEBUG --- stderr --- 20:36:34 DEBUG 20:36:34 INFO 20:36:34 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:34 INFO [loop_until]: OK (rc = 0) 20:36:34 DEBUG --- stdout --- 20:36:34 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:34 DEBUG --- stderr --- 20:36:34 DEBUG 20:36:34 INFO 20:36:34 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:34 INFO [loop_until]: OK (rc = 0) 20:36:34 DEBUG --- stdout --- 20:36:34 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:34 DEBUG --- stderr --- 20:36:34 DEBUG 20:36:34 INFO 20:36:34 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:34 INFO [loop_until]: OK (rc = 0) 20:36:34 DEBUG --- stdout --- 20:36:34 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:36:34 DEBUG --- stderr --- 20:36:34 DEBUG 20:36:34 INFO Initializing monitoring instance threads 20:36:34 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 20:36:34 INFO Starting instance threads 20:36:34 INFO 20:36:34 INFO Thread started 20:36:34 INFO [loop_until]: kubectl --namespace=xlou top node 20:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:34 INFO 20:36:34 INFO Thread started 20:36:34 INFO [loop_until]: kubectl --namespace=xlou top pods 20:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG --- http response --- 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG --- http response --- 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 INFO Thread started 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG ----- output ----- 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG { 20:36:34 DEBUG { 20:36:34 INFO Thread started 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG "status": "success", 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 DEBUG "data": { 20:36:34 DEBUG "data": { 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG --- status code --- 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG "resultType": "vector", 20:36:34 INFO Thread started 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- status code --- 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG "result": [ 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG --- status code --- 20:36:34 INFO Thread started 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG "result": [ 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG { 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG "metric": { 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG --- http response --- 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 INFO Thread started 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG "metric": { 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG --- http response --- 20:36:34 INFO Thread started 20:36:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG }, 20:36:34 DEBUG { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG }, 20:36:34 INFO Thread started 20:36:34 DEBUG { 20:36:34 DEBUG { 20:36:34 DEBUG { 20:36:34 DEBUG { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG { 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG { 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG "value": [ 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG { 20:36:34 DEBUG { 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG "value": [ 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG "status": "success", 20:36:34 INFO Thread started 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG { 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG "data": { 20:36:34 DEBUG 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG 20:36:34 DEBUG "data": { 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG "data": { 20:36:34 INFO Thread started 20:36:34 DEBUG "data": { 20:36:34 DEBUG "data": { 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG "data": { 20:36:34 DEBUG "data": { 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG ----- output ----- 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG "data": { 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG 20:36:34 DEBUG "data": { 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG "data": { 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG 20:36:34 DEBUG "resultType": "vector", 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG "resultType": "vector", 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728194" 20:36:34 DEBUG "resultType": "vector", 20:36:34 INFO Thread started 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG { 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "result": [ 20:36:34 DEBUG 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG { 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "data": { 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG 20:36:34 DEBUG "result": [ 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "result": [ 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728194" 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 INFO Thread started 20:36:34 DEBUG { 20:36:34 INFO [loop_until]: OK (rc = 0) 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "data": { 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG "result": [ 20:36:34 DEBUG { 20:36:34 DEBUG [...] 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG [...] 20:36:34 DEBUG { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG { 20:36:34 DEBUG "metric": { 20:36:34 DEBUG { 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG "status": "success", 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728194" 20:36:34 DEBUG --- stdout --- 20:36:34 INFO Thread started 20:36:34 DEBUG { 20:36:34 DEBUG { 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "data": { 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:34 DEBUG { 20:36:34 DEBUG "metric": { 20:36:34 DEBUG 20:36:34 DEBUG { 20:36:34 DEBUG "data": { 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG { 20:36:34 DEBUG "result": [ 20:36:34 DEBUG { 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG { 20:36:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:36:34 DEBUG "metric": { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG "data": { 20:36:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6306Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 7845Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 11503Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 14704Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5307Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 11909Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 3413Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 65m 0% 11026Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7415Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 92m 0% 6853Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 8064Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 67m 0% 13800Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 12721Mi 21% 20:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728194" 20:36:34 DEBUG "metric": { 20:36:34 INFO Thread started 20:36:34 ERROR [http_cmd]: ERROR 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "instance": "10.142.0.107:9100" 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG "metric": { 20:36:34 DEBUG { 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG 20:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG "metric": { 20:36:34 DEBUG }, 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG --- stderr --- 20:36:34 DEBUG "instance": "10.142.0.107:9100" 20:36:34 INFO All threads has been started 127.0.0.1 - - [21/Apr/2024 20:36:34] "GET /monitoring/start HTTP/1.1" 200 - 20:36:34 ERROR --- status code --- 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG { 20:36:34 DEBUG "result": [ 20:36:34 DEBUG { 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG "instance": "10.142.0.107:9100" 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:34 DEBUG "result": [ 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG "pod": "ds-cts-0" 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "data": { 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG }, 20:36:34 DEBUG "result": [ 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:34 DEBUG 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG }, 20:36:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:36:34 DEBUG }, 20:36:34 DEBUG "metric": { 20:36:34 DEBUG { 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG { 20:36:34 DEBUG }, 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:36:34 DEBUG }, 20:36:34 DEBUG { 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG }, 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:36:34 DEBUG }, 20:36:34 DEBUG "value": [ 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG { 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG "value": [ 20:36:34 ERROR --- http response --- 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "data": { 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "metric": { 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:34 DEBUG "value": [ 20:36:34 DEBUG }, 20:36:34 DEBUG "result": [ 20:36:34 DEBUG { 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG "0.025748998355631214" 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG { 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:36:34 DEBUG }, 20:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "data": { 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG { 20:36:34 DEBUG "status": "success", 20:36:34 INFO [http_cmd]: http status code OK 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG 20:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:36:34 DEBUG "value": [ 20:36:34 DEBUG }, 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG { 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "data": { 20:36:34 DEBUG --- status code --- 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG "data": { 20:36:34 DEBUG 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG { 20:36:34 DEBUG "result": [ 20:36:34 DEBUG 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG 20:36:34 DEBUG [...] 20:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG } 20:36:34 DEBUG [...] 20:36:34 DEBUG [...] 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG [...] 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG { 20:36:34 DEBUG [...] 20:36:34 DEBUG "45169337.570816636" 20:36:34 DEBUG 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG "data": { 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG "result": [ 20:36:34 DEBUG --- http response --- 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "result": [ 20:36:34 DEBUG 20:36:34 DEBUG { 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "pod": "ds-cts-0" 20:36:34 DEBUG "metric": { 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG { 20:36:34 DEBUG 20:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG { 20:36:34 DEBUG 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "pod": "ds-cts-0" 20:36:34 DEBUG } 20:36:34 DEBUG "instance": "10.142.0.43:9100" 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "result": [ 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:34 DEBUG ----- output ----- 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG 20:36:34 DEBUG "data": { 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG [...] 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG ] 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG [...] 20:36:34 DEBUG { 20:36:34 DEBUG 20:36:34 DEBUG "pod": "ds-cts-2" 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG { 20:36:34 DEBUG 20:36:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:36:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:36:34 DEBUG [...] 20:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:34 DEBUG "instance": "10.142.0.43:9100" 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG } 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG [...] 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "status": "success", 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG }, 20:36:34 DEBUG "result": [ 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "instance": "10.142.0.43:9100" 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "data": { 20:36:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "value": [ 20:36:34 DEBUG { 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "pod": "ds-idrepo-2" 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG "value": [ 20:36:34 DEBUG [...] 20:36:34 DEBUG "27182.199999999997" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "pod": "ds-idrepo-0" 20:36:34 DEBUG "pod": "ds-idrepo-2" 20:36:34 DEBUG "pod": "ds-idrepo-2" 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "resultType": "vector", 20:36:34 DEBUG }, 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG "metric": { 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG }, 20:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:34 DEBUG 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG "28197.533333333333" 20:36:34 DEBUG ] 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG }, 20:36:34 DEBUG }, 20:36:34 DEBUG }, 20:36:34 DEBUG "0" 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "result": [ 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "2638577664" 20:36:34 DEBUG "565.9803786099212" 20:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:34 DEBUG 20:36:34 DEBUG "0.0003333333333330302" 20:36:34 DEBUG "instance": "10.142.0.107:9100" 20:36:34 DEBUG "0" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG "0" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG ] 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:34 DEBUG [...] 20:36:34 DEBUG { 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG }, 20:36:34 DEBUG ] 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG "value": [ 20:36:34 DEBUG [...] 20:36:34 DEBUG } 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG 20:36:34 DEBUG "value": [ 20:36:34 DEBUG "0" 20:36:34 DEBUG } 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG }, 20:36:34 DEBUG 20:36:34 DEBUG "metric": { 20:36:34 DEBUG "0.0030325411081957646" 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG "value": [ 20:36:34 DEBUG [...] 20:36:34 DEBUG } 20:36:34 DEBUG "value": [ 20:36:34 DEBUG } 20:36:34 DEBUG "0" 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG "pod": "ds-cts-2" 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG "0" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG "instance": "10.142.0.107:9100" 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG 20:36:34 DEBUG "0" 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG 20:36:34 DEBUG }, 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG "0" 20:36:34 DEBUG ] 20:36:34 DEBUG 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG "0" 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG "pod": "ds-cts-2" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG } 20:36:34 DEBUG [...] 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG }, 20:36:34 DEBUG } 20:36:34 DEBUG "0" 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG "0" 20:36:34 DEBUG } 20:36:34 DEBUG }, 20:36:34 DEBUG } 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG }, 20:36:34 DEBUG [...] 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG "value": [ 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG "value": [ 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG } 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG [...] 20:36:34 DEBUG } 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG "0" 20:36:34 DEBUG ] 20:36:34 DEBUG "pod": "ds-cts-2" 20:36:34 INFO [loop_until]: OK (rc = 0) 20:36:34 DEBUG ] 20:36:34 DEBUG "1" 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG "196.332" 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG }, 20:36:34 DEBUG --- stdout --- 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG 20:36:34 DEBUG ] 20:36:34 DEBUG "instance": "10.142.0.43:9100" 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG "value": [ 20:36:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6m 8711Mi am-767f6b67b5-qcqz2 7m 9050Mi am-767f6b67b5-vbltz 7m 8772Mi ds-cts-0 13m 11353Mi ds-cts-1 25m 1665Mi ds-cts-2 13m 11065Mi ds-idrepo-0 15m 4026Mi ds-idrepo-1 13m 4032Mi ds-idrepo-2 13m 3947Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 1m 54Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-68c5d5b884-p97w5 1m 2421Mi 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG }, 20:36:34 DEBUG ] 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG --- stderr --- 20:36:34 DEBUG ] 20:36:34 DEBUG "instance": "10.142.0.43:9100" 20:36:34 DEBUG ] 20:36:34 DEBUG "value": [ 20:36:34 DEBUG } 20:36:34 DEBUG "0" 20:36:34 DEBUG 20:36:34 DEBUG } 20:36:34 DEBUG }, 20:36:34 DEBUG } 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG "value": [ 20:36:34 DEBUG } 20:36:34 DEBUG "4491525997568" 20:36:34 DEBUG } 20:36:34 DEBUG 1713728194, 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG "0.0003333333333538728" 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG ] 20:36:34 DEBUG } 20:36:34 DEBUG } 20:36:34 DEBUG } 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728254" 20:37:34 INFO 20:37:34 INFO [loop_until]: kubectl --namespace=xlou top node 20:37:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728254" 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728254" 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "pod": "ds-idrepo-1" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG "pod": "ds-idrepo-2" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "0" 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "2.058751569338444" 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "instance": "10.142.0.107:9100" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG "instance": "10.142.0.43:9100" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "0.007299999999958118" 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- status code --- 20:37:34 ERROR [http_cmd]: ERROR 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 ERROR --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 ERROR --- http response --- 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "pod": "ds-idrepo-0" 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "instance": "10.142.0.107:9100" 20:37:34 DEBUG }, 20:37:34 DEBUG "instance": "10.142.0.107:9100" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG }, 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG "value": [ 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG 20:37:34 DEBUG "value": [ 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG 20:37:34 DEBUG --- status code --- 20:37:34 INFO 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 INFO [loop_until]: kubectl --namespace=xlou top pods 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG 20:37:34 DEBUG --- http response --- 20:37:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG [...] 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG { 20:37:34 DEBUG 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "pod": "ds-cts-2" 20:37:34 DEBUG 20:37:34 DEBUG "data": { 20:37:34 DEBUG "instance": "10.142.0.43:9100" 20:37:34 DEBUG "data": { 20:37:34 DEBUG "data": { 20:37:34 DEBUG }, 20:37:34 DEBUG "instance": "10.142.0.43:9100" 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG }, 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "value": [ 20:37:34 DEBUG }, 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "result": [ 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG { 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG "0" 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "metric": { 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG "5731138.666666667" 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG ] 20:37:34 DEBUG "5733635.233333333" 20:37:34 DEBUG "pod": "ds-idrepo-0" 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG ] 20:37:34 DEBUG "pod": "ds-cts-0" 20:37:34 DEBUG "pod": "ds-idrepo-1" 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG }, 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG } 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG "value": [ 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG ] 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG 20:37:34 DEBUG } 20:37:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG 20:37:34 DEBUG } 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG } 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 DEBUG [...] 20:37:34 DEBUG [...] 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728254" 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728254" 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728254" 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG "pod": "ds-idrepo-2" 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG "pod": "ds-cts-2" 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [loop_until]: OK (rc = 0) 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- status code --- 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG }, 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- stdout --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG }, 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG "value": [ 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG }, 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6274Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1973m 6% 7870Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7577m 25% 11540Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3350Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 2151m 7% 14727Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 5311Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3931m 13% 11952Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 5807m 19% 3272Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 7342m 24% 11080Mi 9% gke-xlou-cdm-ds-0f601751-j488 2681m 8% 7441Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 4420m 14% 7385Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 3465m 11% 8085Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 1885m 6% 13843Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 4245m 26% 13266Mi 22% 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG "value": [ 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG "value": [ 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- stderr --- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG 20:37:34 INFO [http_cmd]: http status code OK 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG "0" 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG "0" 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG "0.00004532" 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG --- status code --- 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG ] 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 DEBUG ] 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG ] 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:34 DEBUG { 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG } 20:37:34 DEBUG { 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG } 20:37:34 INFO [loop_until]: OK (rc = 0) 20:37:34 DEBUG --- stdout --- 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG } 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG --- http response --- 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG ] 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG ] 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6179m 8723Mi am-767f6b67b5-qcqz2 3322m 9058Mi am-767f6b67b5-vbltz 5349m 8787Mi ds-cts-0 1286m 11363Mi ds-cts-1 4410m 2441Mi ds-cts-2 2315m 11083Mi ds-idrepo-0 1965m 4026Mi ds-idrepo-1 1631m 4033Mi ds-idrepo-2 2096m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 3756m 2975Mi 20:37:34 DEBUG ] 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "data": { 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG "data": { 20:37:34 DEBUG "data": { 20:37:34 DEBUG "data": { 20:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:34 DEBUG { 20:37:34 DEBUG } 20:37:34 DEBUG "data": { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG } 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG --- stderr --- 20:37:34 DEBUG } 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG ----- output ----- 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG } 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "data": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG } 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG 20:37:34 DEBUG } 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "data": { 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG { 20:37:34 DEBUG "data": { 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG "status": "success", 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG { 20:37:34 DEBUG "result": [ 20:37:34 DEBUG { 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "pod": "ds-cts-0" 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "data": { 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "metric": { 20:37:34 DEBUG { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:34 DEBUG }, 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:34 DEBUG }, 20:37:34 DEBUG "instance": "10.142.0.107:9100" 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:37:34 DEBUG "pod": "ds-idrepo-0" 20:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:34 DEBUG "resultType": "vector", 20:37:34 DEBUG { 20:37:34 DEBUG "pod": "ds-idrepo-1" 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:34 DEBUG "pod": "ds-idrepo-1" 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "pod": "ds-idrepo-0" 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG }, 20:37:34 DEBUG { 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG "result": [ 20:37:34 DEBUG "metric": { 20:37:34 DEBUG }, 20:37:34 DEBUG "pod": "ds-cts-0" 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG "metric": { 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG { 20:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:34 DEBUG "value": [ 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG "metric": { 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG "instance": "10.142.0.107:9100" 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG [...] 20:37:34 DEBUG [...] 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:34 DEBUG 20:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:34 DEBUG 20:37:34 DEBUG "pod": "ds-cts-2" 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:37:34 DEBUG "pod": "ds-idrepo-2" 20:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:34 DEBUG [...] 20:37:34 DEBUG 20:37:34 DEBUG "pod": "ds-idrepo-2" 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "instance": "10.142.0.43:9100" 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG }, 20:37:34 DEBUG 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG "pod": "ds-cts-2" 20:37:34 DEBUG 20:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG }, 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 20:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:34 DEBUG "value": [ 20:37:34 DEBUG }, 20:37:34 DEBUG "pod": "ds-idrepo-2" 20:37:34 DEBUG }, 20:37:34 DEBUG "0" 20:37:34 DEBUG "1050.4" 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "1051.6666666666667" 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG 20:37:34 DEBUG }, 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG }, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG "0" 20:37:34 DEBUG ] 20:37:34 DEBUG "0" 20:37:34 DEBUG "0.005333333333328484" 20:37:34 DEBUG }, 20:37:34 DEBUG "3187101696" 20:37:34 DEBUG "0" 20:37:34 DEBUG "1051.5" 20:37:34 DEBUG "instance": "10.142.0.43:9100" 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "0" 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "value": [ 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG "value": [ 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG }, 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG ] 20:37:34 DEBUG "0" 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG "0" 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG "value": [ 20:37:34 DEBUG "196.333" 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG "0" 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG "15742.5" 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG 1713728254, 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG "4491575550976" 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG ] 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:37:34 DEBUG } 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728314" 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "instance": "10.142.0.107:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "instance": "10.142.0.43:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "0.10770000000005286" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "11.762253654901643" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "instance": "10.142.0.107:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "instance": "10.142.0.43:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "79851772.63333333" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "pod": "ds-idrepo-0" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "pod": "ds-cts-2" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "0" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "286130764.48489666" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728314" 20:38:34 ERROR [http_cmd]: ERROR 20:38:34 ERROR --- status code --- 20:38:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:38:34 ERROR --- http response --- 20:38:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:38:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728314" 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "instance": "10.142.0.107:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "instance": "10.142.0.43:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "79852884.36666667" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728314" 20:38:34 INFO 20:38:34 INFO [loop_until]: kubectl --namespace=xlou top node 20:38:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG ----- output ----- 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG --- http response --- 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 INFO 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:34 DEBUG { 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG --- status code --- 20:38:34 INFO [loop_until]: kubectl --namespace=xlou top pods 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG ----- output ----- 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:38:34 DEBUG { 20:38:34 DEBUG "data": { 20:38:34 DEBUG { 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG "data": { 20:38:34 DEBUG "result": [ 20:38:34 DEBUG "data": { 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:34 DEBUG "status": "success", 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG "result": [ 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "result": [ 20:38:34 DEBUG ----- output ----- 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG "data": { 20:38:34 DEBUG --- status code --- 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG { 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG "pod": "ds-idrepo-0" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG { 20:38:34 DEBUG { 20:38:34 DEBUG "resultType": "vector", 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728314" 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG "metric": { 20:38:34 DEBUG }, 20:38:34 DEBUG "metric": { 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "result": [ 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- http response --- 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG "pod": "ds-cts-0" 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "pod": "ds-idrepo-0" 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG "data": { 20:38:34 DEBUG { 20:38:34 DEBUG --- status code --- 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG }, 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "metric": { 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG "value": [ 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG 20:38:34 DEBUG "value": [ 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG "result": [ 20:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG { 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG { 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG { 20:38:34 INFO [http_cmd]: http status code OK 20:38:34 DEBUG }, 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG { 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG "metric": { 20:38:34 DEBUG --- status code --- 20:38:34 DEBUG "value": [ 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG { 20:38:34 DEBUG "data": { 20:38:34 DEBUG "data": { 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG { 20:38:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:34 DEBUG 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG [...] 20:38:34 DEBUG { 20:38:34 DEBUG "data": { 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG }, 20:38:34 DEBUG --- http response --- 20:38:34 DEBUG 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG { 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "data": { 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG 20:38:34 DEBUG "data": { 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "pod": "ds-idrepo-2" 20:38:34 DEBUG 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG ----- output ----- 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG { 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "data": { 20:38:34 DEBUG }, 20:38:34 DEBUG 20:38:34 DEBUG "result": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG "pod": "ds-cts-2" 20:38:34 DEBUG "result": [ 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "data": { 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "pod": "ds-idrepo-2" 20:38:34 DEBUG { 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "status": "success", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:34 DEBUG }, 20:38:34 DEBUG }, 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG }, 20:38:34 DEBUG { 20:38:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "result": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG }, 20:38:34 DEBUG "metric": { 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG "data": { 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "pod": "ds-cts-0" 20:38:34 DEBUG "pod": "ds-cts-0" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "metric": { 20:38:34 DEBUG }, 20:38:34 DEBUG "result": [ 20:38:34 DEBUG { 20:38:34 DEBUG "0" 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:34 DEBUG 20:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:34 DEBUG "resultType": "vector", 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "pod": "ds-idrepo-0" 20:38:34 DEBUG "value": [ 20:38:34 DEBUG { 20:38:34 DEBUG "metric": { 20:38:34 DEBUG ] 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG }, 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG "result": [ 20:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:34 DEBUG }, 20:38:34 DEBUG "0" 20:38:34 DEBUG }, 20:38:34 DEBUG 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "instance": "10.142.0.107:9100" 20:38:34 DEBUG } 20:38:34 DEBUG "0" 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG "value": [ 20:38:34 DEBUG { 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG ] 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG "instance": "10.142.0.107:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG ] 20:38:34 DEBUG ] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "metric": { 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:38:34 DEBUG "196.33350000000002" 20:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG ] 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG [...] 20:38:34 DEBUG }, 20:38:34 DEBUG ] 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG "value": [ 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG ] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:34 DEBUG 20:38:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG "3273650176" 20:38:34 DEBUG } 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 INFO [loop_until]: OK (rc = 0) 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:38:34 DEBUG [...] 20:38:34 DEBUG 20:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG [...] 20:38:34 DEBUG --- stdout --- 20:38:34 DEBUG 20:38:34 DEBUG "pod": "ds-cts-2" 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "pod": "ds-cts-2" 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG }, 20:38:34 DEBUG } 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17527m 8730Mi am-767f6b67b5-qcqz2 17420m 8934Mi am-767f6b67b5-vbltz 18660m 8796Mi ds-cts-0 9530m 11404Mi ds-cts-1 10130m 4613Mi ds-cts-2 9969m 10869Mi ds-idrepo-0 8456m 4026Mi ds-idrepo-1 8367m 4033Mi ds-idrepo-2 8393m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 1m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11765m 3069Mi 20:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:34 DEBUG }, 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG }, 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG 20:38:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:38:34 DEBUG 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 20:38:34 INFO [loop_until]: OK (rc = 0) 20:38:34 DEBUG 20:38:34 DEBUG "value": [ 20:38:34 DEBUG ] 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG --- stderr --- 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "3329.266666666667" 20:38:34 DEBUG "value": [ 20:38:34 DEBUG "1501.2369811551268" 20:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:34 DEBUG }, 20:38:34 DEBUG "pod": "ds-idrepo-2" 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG 20:38:34 DEBUG --- stdout --- 20:38:34 DEBUG "instance": "10.142.0.43:9100" 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG } 20:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:34 DEBUG 20:38:34 DEBUG 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG ] 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG ] 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG }, 20:38:34 DEBUG "0" 20:38:34 DEBUG "instance": "10.142.0.43:9100" 20:38:34 DEBUG }, 20:38:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8763m 29% 7863Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17543m 58% 11528Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9927m 33% 14795Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5295Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17926m 60% 11813Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19320m 64% 3260Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 20185m 67% 11077Mi 9% gke-xlou-cdm-ds-0f601751-j488 9212m 30% 7443Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 10049m 33% 10235Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 9087m 30% 8075Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 9927m 33% 13649Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 12153m 76% 13354Mi 22% 20:38:34 DEBUG "3325.4666666666667" 20:38:34 DEBUG } 20:38:34 DEBUG }, 20:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "0" 20:38:34 DEBUG } 20:38:34 DEBUG "0.00012265999999999997" 20:38:34 DEBUG } 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG ] 20:38:34 DEBUG }, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG --- stderr --- 20:38:34 DEBUG ] 20:38:34 DEBUG "value": [ 20:38:34 DEBUG }, 20:38:34 DEBUG "0" 20:38:34 DEBUG ] 20:38:34 DEBUG ] 20:38:34 DEBUG ] 20:38:34 DEBUG ] 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "17.704441342659567" 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG } 20:38:34 DEBUG "value": [ 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG 20:38:34 DEBUG } 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "value": [ 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG "3326.5" 20:38:34 DEBUG ] 20:38:34 DEBUG "0" 20:38:34 DEBUG ] 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG "4493945123840" 20:38:34 DEBUG ] 20:38:34 DEBUG "0" 20:38:34 DEBUG 1713728314, 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG "0.06499999999999961" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG "179016.5" 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG ] 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:38:34 DEBUG } 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "11.778836704993585" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728374" 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "ds-idrepo-0" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "ds-cts-2" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "0" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728374" 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "instance": "10.142.0.107:9100" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "instance": "10.142.0.43:9100" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "0.058900000000047234" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 ERROR [http_cmd]: ERROR 20:39:34 ERROR --- status code --- 20:39:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:39:34 ERROR --- http response --- 20:39:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:39:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "instance": "10.142.0.107:9100" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "instance": "10.142.0.43:9100" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "79971201.96666667" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "38014090.774285465" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "instance": "10.142.0.107:9100" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "instance": "10.142.0.43:9100" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "79972164.13333333" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG { 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "ds-cts-0" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "ds-cts-2" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "0" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "196.334" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [loop_until]: kubectl --namespace=xlou top pods 20:39:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "ds-idrepo-0" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "ds-idrepo-2" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "0" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "ds-idrepo-0" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "ds-idrepo-2" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "0" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "3330.733333333333" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:34 DEBUG ----- output ----- 20:39:34 DEBUG { 20:39:34 DEBUG "status": "success", 20:39:34 DEBUG "data": { 20:39:34 DEBUG "resultType": "vector", 20:39:34 DEBUG "result": [ 20:39:34 DEBUG { 20:39:34 DEBUG "metric": { 20:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG [...] 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG 20:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:34 DEBUG }, 20:39:34 DEBUG "value": [ 20:39:34 DEBUG 1713728374, 20:39:34 DEBUG "0" 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG ] 20:39:34 DEBUG } 20:39:34 DEBUG } 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728374" 20:39:34 INFO 20:39:34 INFO [loop_until]: kubectl --namespace=xlou top node 20:39:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:34 INFO [http_cmd]: http status code OK 20:39:34 DEBUG --- status code --- 20:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:34 DEBUG --- http response --- 20:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728374" 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728375" 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728375" 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728375" 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG { 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG "data": { 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728375" 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728375" 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728375" 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728375" 20:39:35 DEBUG "resultType": "vector", 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728375" 20:39:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728375" 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG "result": [ 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG { 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG --- http response --- 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG }, 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG "value": [ 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG { 20:39:35 DEBUG 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:35 INFO [http_cmd]: http status code OK 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:35 DEBUG { 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG --- status code --- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG "data": { 20:39:35 DEBUG 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG { 20:39:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:35 DEBUG { 20:39:35 DEBUG "data": { 20:39:35 DEBUG "data": { 20:39:35 DEBUG "data": { 20:39:35 DEBUG "data": { 20:39:35 DEBUG { 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:35 DEBUG "data": { 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG [...] 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG --- http response --- 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "result": [ 20:39:35 DEBUG 20:39:35 DEBUG "data": { 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:35 DEBUG "data": { 20:39:35 DEBUG "result": [ 20:39:35 DEBUG "result": [ 20:39:35 DEBUG "result": [ 20:39:35 DEBUG "result": [ 20:39:35 DEBUG "data": { 20:39:35 DEBUG { 20:39:35 DEBUG "result": [ 20:39:35 DEBUG { 20:39:35 DEBUG 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG ----- output ----- 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG { 20:39:35 DEBUG "metric": { 20:39:35 DEBUG 20:39:35 DEBUG "result": [ 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG "result": [ 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "result": [ 20:39:35 DEBUG "data": { 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:39:35 DEBUG { 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG "status": "success", 20:39:35 DEBUG { 20:39:35 DEBUG "pod": "ds-idrepo-0" 20:39:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:39:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:35 DEBUG { 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "data": { 20:39:35 DEBUG "data": { 20:39:35 DEBUG "metric": { 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "result": [ 20:39:35 DEBUG }, 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "pod": "ds-cts-0" 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "resultType": "vector", 20:39:35 DEBUG "pod": "ds-cts-0" 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:35 DEBUG { 20:39:35 DEBUG "value": [ 20:39:35 DEBUG 20:39:35 DEBUG 1713728374, 20:39:35 DEBUG }, 20:39:35 DEBUG "result": [ 20:39:35 DEBUG "result": [ 20:39:35 DEBUG }, 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG }, 20:39:35 DEBUG "metric": { 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG "0" 20:39:35 DEBUG "value": [ 20:39:35 DEBUG { 20:39:35 DEBUG { 20:39:35 DEBUG "value": [ 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "instance": "10.142.0.107:9100" 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG ] 20:39:35 DEBUG 20:39:35 DEBUG "metric": { 20:39:35 DEBUG "metric": { 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG }, 20:39:35 DEBUG 20:39:35 DEBUG [...] 20:39:35 DEBUG } 20:39:35 DEBUG 20:39:35 DEBUG "pod": "ds-idrepo-0" 20:39:35 DEBUG "instance": "10.142.0.107:9100" 20:39:35 DEBUG 20:39:35 DEBUG [...] 20:39:35 DEBUG [...] 20:39:35 DEBUG [...] 20:39:35 DEBUG [...] 20:39:35 DEBUG 20:39:35 DEBUG "value": [ 20:39:35 DEBUG [...] 20:39:35 DEBUG 20:39:35 DEBUG ] 20:39:35 DEBUG 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG } 20:39:35 DEBUG [...] 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG [...] 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG [...] 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG } 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:39:35 DEBUG }, 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:39:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:39:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:35 DEBUG 20:39:35 DEBUG [...] 20:39:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG [...] 20:39:35 DEBUG 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG }, 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG }, 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "pod": "ds-cts-2" 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG "pod": "ds-cts-2" 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:35 DEBUG }, 20:39:35 DEBUG "value": [ 20:39:35 DEBUG 1713728374, 20:39:35 DEBUG }, 20:39:35 DEBUG [...] 20:39:35 DEBUG 20:39:35 DEBUG }, 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG 1713728374, 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG 20:39:35 DEBUG "value": [ 20:39:35 DEBUG 1713728374, 20:39:35 DEBUG "476.3760067455587" 20:39:35 DEBUG "value": [ 20:39:35 DEBUG 20:39:35 DEBUG 20:39:35 DEBUG "value": [ 20:39:35 DEBUG "0.002943942081238842" 20:39:35 DEBUG "3330.366666666667" 20:39:35 DEBUG "3322613760" 20:39:35 DEBUG "0" 20:39:35 DEBUG 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG "3330.4" 20:39:35 DEBUG ] 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG 20:39:35 DEBUG "pod": "ds-idrepo-2" 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG "instance": "10.142.0.43:9100" 20:39:35 DEBUG "379027" 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG "0" 20:39:35 DEBUG 20:39:35 DEBUG }, 20:39:35 DEBUG "0.000023215555555555553" 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG }, 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG "instance": "10.142.0.43:9100" 20:39:35 DEBUG "value": [ 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG "value": [ 20:39:35 DEBUG } 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG }, 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG "0" 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG "0.04033333333332506" 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG "value": [ 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG 1713728375, 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG "4497597199360" 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 DEBUG ] 20:39:35 DEBUG } 20:39:35 DEBUG } 20:39:35 INFO [loop_until]: OK (rc = 0) 20:39:35 DEBUG --- stdout --- 20:39:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17690m 8734Mi am-767f6b67b5-qcqz2 17561m 8941Mi am-767f6b67b5-vbltz 19305m 8801Mi ds-cts-0 10143m 11414Mi ds-cts-1 9870m 7579Mi ds-cts-2 10152m 10876Mi ds-idrepo-0 8426m 4026Mi ds-idrepo-1 8159m 4033Mi ds-idrepo-2 8540m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11725m 3078Mi 20:39:35 DEBUG --- stderr --- 20:39:35 DEBUG 20:39:35 INFO [loop_until]: OK (rc = 0) 20:39:35 DEBUG --- stdout --- 20:39:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6292Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8693m 29% 7859Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 16778m 56% 11548Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3351Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 10211m 34% 14794Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5293Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 18001m 60% 11837Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19476m 65% 3244Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 20019m 67% 11081Mi 9% gke-xlou-cdm-ds-0f601751-j488 9224m 30% 7434Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 10011m 33% 13140Mi 11% gke-xlou-cdm-ds-0f601751-nj6q 9125m 30% 8074Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 10204m 34% 13651Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11597m 72% 13360Mi 22% 20:39:35 DEBUG --- stderr --- 20:39:35 DEBUG 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728434" 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728434" 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728434" 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728434" 20:40:34 INFO [http_cmd]: http status code OK 20:40:34 DEBUG --- status code --- 20:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:34 DEBUG --- http response --- 20:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:34 DEBUG ----- output ----- 20:40:34 DEBUG { 20:40:34 DEBUG "status": "success", 20:40:34 DEBUG "data": { 20:40:34 DEBUG "resultType": "vector", 20:40:34 DEBUG "result": [ 20:40:34 DEBUG { 20:40:34 DEBUG "metric": { 20:40:34 DEBUG "pod": "ds-idrepo-0" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG [...] 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG "pod": "ds-cts-2" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 1713728434, 20:40:34 DEBUG "0" 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG } 20:40:34 INFO [http_cmd]: http status code OK 20:40:34 DEBUG --- status code --- 20:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:34 DEBUG --- http response --- 20:40:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:40:34 INFO [http_cmd]: http status code OK 20:40:34 DEBUG --- status code --- 20:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:34 DEBUG --- http response --- 20:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:34 DEBUG ----- output ----- 20:40:34 INFO [http_cmd]: http status code OK 20:40:34 DEBUG { 20:40:34 DEBUG --- status code --- 20:40:34 DEBUG "status": "success", 20:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:34 DEBUG "data": { 20:40:34 DEBUG --- http response --- 20:40:34 DEBUG "resultType": "vector", 20:40:34 DEBUG "result": [ 20:40:34 DEBUG { 20:40:34 DEBUG "metric": { 20:40:34 DEBUG "pod": "ds-idrepo-0" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG [...] 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 1713728434, 20:40:34 DEBUG "0" 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG } 20:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:34 DEBUG ----- output ----- 20:40:34 DEBUG { 20:40:34 DEBUG "status": "success", 20:40:34 DEBUG "data": { 20:40:34 DEBUG "resultType": "vector", 20:40:34 DEBUG "result": [ 20:40:34 DEBUG { 20:40:34 DEBUG "metric": { 20:40:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG [...] 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 1713728434, 20:40:34 DEBUG "11.679528482672646" 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG } 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728434" 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728434" 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728434" 20:40:34 ERROR [http_cmd]: ERROR 20:40:34 ERROR --- status code --- 20:40:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:40:34 ERROR --- http response --- 20:40:34 INFO [http_cmd]: http status code OK 20:40:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:40:34 DEBUG --- status code --- 20:40:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:34 DEBUG --- http response --- 20:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:34 DEBUG ----- output ----- 20:40:34 DEBUG { 20:40:34 DEBUG "status": "success", 20:40:34 DEBUG "data": { 20:40:34 DEBUG "resultType": "vector", 20:40:34 DEBUG "result": [ 20:40:34 DEBUG { 20:40:34 DEBUG "metric": { 20:40:34 DEBUG "instance": "10.142.0.107:9100" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG [...] 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG "instance": "10.142.0.43:9100" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 1713728434, 20:40:34 DEBUG "0.06306666666671579" 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG } 20:40:34 INFO [http_cmd]: http status code OK 20:40:34 DEBUG --- status code --- 20:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:34 DEBUG --- http response --- 20:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:34 DEBUG ----- output ----- 20:40:34 DEBUG { 20:40:34 DEBUG "status": "success", 20:40:34 DEBUG "data": { 20:40:34 DEBUG "resultType": "vector", 20:40:34 DEBUG "result": [ 20:40:34 DEBUG { 20:40:34 DEBUG "metric": { 20:40:34 DEBUG "instance": "10.142.0.107:9100" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG [...] 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG "instance": "10.142.0.43:9100" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 1713728434, 20:40:34 DEBUG "81502843.4" 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG } 20:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728434" 20:40:34 INFO [http_cmd]: http status code OK 20:40:34 DEBUG --- status code --- 20:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:34 DEBUG --- http response --- 20:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:34 DEBUG ----- output ----- 20:40:34 DEBUG { 20:40:34 DEBUG "status": "success", 20:40:34 DEBUG "data": { 20:40:34 DEBUG "resultType": "vector", 20:40:34 DEBUG "result": [ 20:40:34 DEBUG { 20:40:34 DEBUG "metric": { 20:40:34 DEBUG "instance": "10.142.0.107:9100" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG [...] 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG 20:40:34 DEBUG "instance": "10.142.0.43:9100" 20:40:34 DEBUG }, 20:40:34 DEBUG "value": [ 20:40:34 DEBUG 1713728434, 20:40:34 DEBUG "81504534.93333334" 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG ] 20:40:34 DEBUG } 20:40:34 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "196.334" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "3335.9333333333334" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "0" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "ds-cts-0" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "ds-cts-2" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "0" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "ds-idrepo-0" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "ds-idrepo-2" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "0" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "ds-idrepo-0" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "ds-idrepo-2" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "0" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "instance": "10.142.0.107:9100" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "instance": "10.142.0.43:9100" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "0.041000000000004394" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "1045.5" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728435" 20:40:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728435" 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG --- status code --- 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG { 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:35 DEBUG { 20:40:35 DEBUG { 20:40:35 DEBUG "data": { 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG { 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG { 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "status": "success", 20:40:35 INFO [http_cmd]: http status code OK 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG { 20:40:35 DEBUG "data": { 20:40:35 DEBUG "data": { 20:40:35 DEBUG --- status code --- 20:40:35 DEBUG "result": [ 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "data": { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "data": { 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:35 DEBUG { 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "data": { 20:40:35 DEBUG "result": [ 20:40:35 DEBUG "result": [ 20:40:35 DEBUG --- http response --- 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG "result": [ 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "result": [ 20:40:35 DEBUG "result": [ 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG { 20:40:35 DEBUG { 20:40:35 DEBUG "pod": "ds-idrepo-0" 20:40:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG { 20:40:35 DEBUG "result": [ 20:40:35 DEBUG { 20:40:35 DEBUG { 20:40:35 DEBUG "result": [ 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG }, 20:40:35 DEBUG ----- output ----- 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG { 20:40:35 DEBUG "pod": "ds-idrepo-0" 20:40:35 DEBUG "pod": "ds-cts-0" 20:40:35 DEBUG "value": [ 20:40:35 DEBUG { 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:40:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:35 DEBUG "metric": { 20:40:35 DEBUG "pod": "ds-idrepo-0" 20:40:35 DEBUG "pod": "ds-cts-0" 20:40:35 DEBUG "metric": { 20:40:35 DEBUG }, 20:40:35 DEBUG }, 20:40:35 DEBUG 20:40:35 DEBUG "status": "success", 20:40:35 DEBUG "pod": "ds-idrepo-0" 20:40:35 DEBUG }, 20:40:35 DEBUG }, 20:40:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:35 DEBUG }, 20:40:35 DEBUG }, 20:40:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG "data": { 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "value": [ 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "value": [ 20:40:35 DEBUG }, 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "resultType": "vector", 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG "result": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG { 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG "metric": { 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "instance": "10.142.0.107:9100" 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG [...] 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "ds-idrepo-1" 20:40:35 DEBUG }, 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "pod": "ds-idrepo-1" 20:40:35 DEBUG "pod": "ds-cts-2" 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 20:40:35 DEBUG 20:40:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:40:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:35 DEBUG 20:40:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:40:35 DEBUG "pod": "ds-cts-2" 20:40:35 DEBUG 20:40:35 DEBUG }, 20:40:35 DEBUG }, 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG 20:40:35 DEBUG "pod": "ds-idrepo-2" 20:40:35 DEBUG }, 20:40:35 DEBUG }, 20:40:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:35 DEBUG }, 20:40:35 DEBUG }, 20:40:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "0" 20:40:35 DEBUG 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "value": [ 20:40:35 DEBUG }, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "value": [ 20:40:35 DEBUG }, 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG ] 20:40:35 DEBUG [...] 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "value": [ 20:40:35 DEBUG "2467.425896696239" 20:40:35 DEBUG "0" 20:40:35 DEBUG } 20:40:35 DEBUG 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG "3329839104" 20:40:35 DEBUG "3335.5666666666666" 20:40:35 DEBUG ] 20:40:35 DEBUG "0" 20:40:35 DEBUG "0.000822662222222222" 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG ] 20:40:35 DEBUG ] 20:40:35 DEBUG ] 20:40:35 DEBUG 20:40:35 DEBUG "0" 20:40:35 DEBUG ] 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG ] 20:40:35 DEBUG "3338.366666666667" 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG "563524" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG ] 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG "instance": "10.142.0.43:9100" 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG }, 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG "value": [ 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 DEBUG 1713728435, 20:40:35 DEBUG } 20:40:35 DEBUG "4501315902464" 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG ] 20:40:35 DEBUG } 20:40:35 DEBUG } 20:40:35 INFO 20:40:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:40:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:40:35 INFO 20:40:35 INFO [loop_until]: kubectl --namespace=xlou top node 20:40:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:40:35 INFO [loop_until]: OK (rc = 0) 20:40:35 DEBUG --- stdout --- 20:40:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 19313m 8744Mi am-767f6b67b5-qcqz2 19433m 8951Mi am-767f6b67b5-vbltz 21657m 8823Mi ds-cts-0 13250m 11415Mi ds-cts-1 11588m 10267Mi ds-cts-2 15066m 10887Mi ds-idrepo-0 8613m 4026Mi ds-idrepo-1 8439m 4021Mi ds-idrepo-2 8974m 3947Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11924m 3135Mi 20:40:35 DEBUG --- stderr --- 20:40:35 DEBUG 20:40:35 INFO [loop_until]: OK (rc = 0) 20:40:35 DEBUG --- stdout --- 20:40:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6296Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8799m 29% 7867Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 18467m 61% 11535Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3360Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 13958m 46% 14727Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5286Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 19221m 64% 11847Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19565m 65% 3244Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 21982m 73% 11115Mi 9% gke-xlou-cdm-ds-0f601751-j488 9655m 32% 7451Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 15291m 51% 15810Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 9537m 31% 8077Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 13817m 46% 13597Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11912m 74% 13364Mi 22% 20:40:35 DEBUG --- stderr --- 20:40:35 DEBUG 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728494" 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728494" 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728494" 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728494" 20:41:34 ERROR [http_cmd]: ERROR 20:41:34 ERROR --- status code --- 20:41:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:41:34 ERROR --- http response --- 20:41:34 INFO [http_cmd]: http status code OK 20:41:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:41:34 DEBUG --- status code --- 20:41:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:34 DEBUG --- http response --- 20:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:41:34 DEBUG ----- output ----- 20:41:34 DEBUG { 20:41:34 DEBUG "status": "success", 20:41:34 DEBUG "data": { 20:41:34 DEBUG "resultType": "vector", 20:41:34 DEBUG "result": [ 20:41:34 DEBUG { 20:41:34 DEBUG "metric": { 20:41:34 DEBUG "pod": "ds-idrepo-0" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG [...] 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 1713728494, 20:41:34 DEBUG "0" 20:41:34 DEBUG ] 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728494" 20:41:34 DEBUG } 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG } 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728494" 20:41:34 INFO [http_cmd]: http status code OK 20:41:34 DEBUG --- status code --- 20:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:34 DEBUG --- http response --- 20:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:34 DEBUG ----- output ----- 20:41:34 DEBUG { 20:41:34 DEBUG "status": "success", 20:41:34 DEBUG "data": { 20:41:34 DEBUG "resultType": "vector", 20:41:34 DEBUG "result": [ 20:41:34 DEBUG { 20:41:34 DEBUG "metric": { 20:41:34 DEBUG "pod": "ds-idrepo-0" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG [...] 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG "pod": "ds-cts-2" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 1713728494, 20:41:34 DEBUG "0" 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG } 20:41:34 INFO [http_cmd]: http status code OK 20:41:34 DEBUG --- status code --- 20:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:34 DEBUG --- http response --- 20:41:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:41:34 INFO [http_cmd]: http status code OK 20:41:34 DEBUG --- status code --- 20:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:34 DEBUG --- http response --- 20:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:34 DEBUG ----- output ----- 20:41:34 DEBUG { 20:41:34 DEBUG "status": "success", 20:41:34 DEBUG "data": { 20:41:34 DEBUG "resultType": "vector", 20:41:34 DEBUG "result": [ 20:41:34 DEBUG { 20:41:34 DEBUG "metric": { 20:41:34 DEBUG "instance": "10.142.0.107:9100" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG [...] 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG "instance": "10.142.0.43:9100" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 1713728494, 20:41:34 DEBUG "100333098.7" 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG } 20:41:34 INFO [http_cmd]: http status code OK 20:41:34 DEBUG --- status code --- 20:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:34 DEBUG --- http response --- 20:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:34 DEBUG ----- output ----- 20:41:34 DEBUG { 20:41:34 DEBUG "status": "success", 20:41:34 DEBUG "data": { 20:41:34 DEBUG "resultType": "vector", 20:41:34 DEBUG "result": [ 20:41:34 DEBUG { 20:41:34 DEBUG "metric": { 20:41:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG [...] 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 1713728494, 20:41:34 DEBUG "11.719252445221727" 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG } 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728494" 20:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728494" 20:41:34 INFO [http_cmd]: http status code OK 20:41:34 DEBUG --- status code --- 20:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:34 DEBUG --- http response --- 20:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:34 DEBUG ----- output ----- 20:41:34 DEBUG { 20:41:34 DEBUG "status": "success", 20:41:34 DEBUG "data": { 20:41:34 DEBUG "resultType": "vector", 20:41:34 DEBUG "result": [ 20:41:34 DEBUG { 20:41:34 DEBUG "metric": { 20:41:34 DEBUG "instance": "10.142.0.107:9100" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG [...] 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG "instance": "10.142.0.43:9100" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 1713728494, 20:41:34 DEBUG "0.17683333333330134" 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG } 20:41:34 INFO [http_cmd]: http status code OK 20:41:34 DEBUG --- status code --- 20:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:34 DEBUG --- http response --- 20:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:34 DEBUG ----- output ----- 20:41:34 DEBUG { 20:41:34 DEBUG "status": "success", 20:41:34 DEBUG "data": { 20:41:34 DEBUG "resultType": "vector", 20:41:34 DEBUG "result": [ 20:41:34 DEBUG { 20:41:34 DEBUG "metric": { 20:41:34 DEBUG "instance": "10.142.0.107:9100" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG [...] 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG 20:41:34 DEBUG "instance": "10.142.0.43:9100" 20:41:34 DEBUG }, 20:41:34 DEBUG "value": [ 20:41:34 DEBUG 1713728494, 20:41:34 DEBUG "100335478.93333334" 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG ] 20:41:34 DEBUG } 20:41:34 DEBUG } 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "pod": "ds-cts-0" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "pod": "ds-cts-2" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "0" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "196.334" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG { 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG "metric": { 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:35 DEBUG { 20:41:35 DEBUG }, 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "data": { 20:41:35 DEBUG 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG 20:41:35 DEBUG "result": [ 20:41:35 DEBUG 20:41:35 DEBUG { 20:41:35 DEBUG [...] 20:41:35 DEBUG "metric": { 20:41:35 DEBUG 20:41:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:35 DEBUG 20:41:35 DEBUG }, 20:41:35 DEBUG 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:35 DEBUG 20:41:35 DEBUG }, 20:41:35 DEBUG 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG [...] 20:41:35 DEBUG "0" 20:41:35 DEBUG 20:41:35 DEBUG ] 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG 20:41:35 DEBUG ] 20:41:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:35 DEBUG } 20:41:35 DEBUG }, 20:41:35 DEBUG } 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "3341.133333333333" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "pod": "ds-idrepo-0" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "pod": "ds-idrepo-2" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "0" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "pod": "ds-idrepo-0" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "pod": "ds-idrepo-2" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "0" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728495" 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "instance": "10.142.0.107:9100" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "instance": "10.142.0.43:9100" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "0.04933333333333204" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "pod": "ds-idrepo-0" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "pod": "ds-idrepo-2" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "0" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "3321.7" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "data": { 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "result": [ 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "instance": "10.142.0.107:9100" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG [...] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "instance": "10.142.0.43:9100" 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG "4505314018304" 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728495" 20:41:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728495" 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:41:35 DEBUG ----- output ----- 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG ----- output ----- 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG { 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG { 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG "data": { 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG "data": { 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG "resultType": "vector", 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:35 DEBUG "result": [ 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG "result": [ 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG { 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG { 20:41:35 DEBUG { 20:41:35 DEBUG { 20:41:35 DEBUG { 20:41:35 DEBUG { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:35 DEBUG "data": { 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:35 DEBUG "data": { 20:41:35 DEBUG "pod": "ds-idrepo-0" 20:41:35 DEBUG "data": { 20:41:35 DEBUG "data": { 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG }, 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG }, 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG "result": [ 20:41:35 INFO [http_cmd]: http status code OK 20:41:35 DEBUG "result": [ 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "result": [ 20:41:35 DEBUG "result": [ 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG { 20:41:35 DEBUG 20:41:35 DEBUG { 20:41:35 DEBUG --- status code --- 20:41:35 DEBUG { 20:41:35 DEBUG 20:41:35 DEBUG { 20:41:35 DEBUG { 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG 20:41:35 DEBUG "metric": { 20:41:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:35 DEBUG "metric": { 20:41:35 DEBUG 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:41:35 DEBUG "data": { 20:41:35 DEBUG [...] 20:41:35 DEBUG "pod": "ds-cts-0" 20:41:35 DEBUG --- http response --- 20:41:35 DEBUG "pod": "ds-idrepo-0" 20:41:35 DEBUG 20:41:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:35 DEBUG "pod": "ds-cts-0" 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG 20:41:35 DEBUG }, 20:41:35 DEBUG }, 20:41:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:35 DEBUG [...] 20:41:35 DEBUG }, 20:41:35 DEBUG }, 20:41:35 DEBUG { 20:41:35 DEBUG "result": [ 20:41:35 DEBUG 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "value": [ 20:41:35 DEBUG ----- output ----- 20:41:35 DEBUG 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG { 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG { 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "data": { 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "status": "success", 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:41:35 DEBUG }, 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "data": { 20:41:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "result": [ 20:41:35 DEBUG }, 20:41:35 DEBUG "value": [ 20:41:35 DEBUG [...] 20:41:35 DEBUG [...] 20:41:35 DEBUG "resultType": "vector", 20:41:35 DEBUG }, 20:41:35 DEBUG [...] 20:41:35 DEBUG [...] 20:41:35 DEBUG { 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "result": [ 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "metric": { 20:41:35 DEBUG 20:41:35 DEBUG "600162" 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG { 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "pod": "ds-idrepo-0" 20:41:35 DEBUG 20:41:35 DEBUG ] 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "metric": { 20:41:35 DEBUG "0" 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG }, 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG "pod": "ds-cts-2" 20:41:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:41:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:41:35 DEBUG ] 20:41:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:35 DEBUG "pod": "ds-cts-2" 20:41:35 DEBUG "value": [ 20:41:35 DEBUG [...] 20:41:35 DEBUG ] 20:41:35 DEBUG }, 20:41:35 DEBUG }, 20:41:35 DEBUG }, 20:41:35 DEBUG } 20:41:35 DEBUG }, 20:41:35 DEBUG }, 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "value": [ 20:41:35 DEBUG ] 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "value": [ 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG 20:41:35 DEBUG 20:41:35 DEBUG "0" 20:41:35 DEBUG "0" 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG "3339.5666666666666" 20:41:35 DEBUG ] 20:41:35 DEBUG [...] 20:41:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:41:35 DEBUG ] 20:41:35 DEBUG ] 20:41:35 DEBUG 20:41:35 DEBUG "0.36093333333333333" 20:41:35 DEBUG } 20:41:35 DEBUG 20:41:35 DEBUG }, 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 DEBUG [...] 20:41:35 DEBUG ] 20:41:35 DEBUG ] 20:41:35 DEBUG 20:41:35 DEBUG "value": [ 20:41:35 DEBUG ] 20:41:35 DEBUG ] 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 DEBUG 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 DEBUG 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:41:35 DEBUG "3338.633333333333" 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 DEBUG 20:41:35 DEBUG } 20:41:35 DEBUG }, 20:41:35 DEBUG ] 20:41:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:41:35 DEBUG } 20:41:35 DEBUG "value": [ 20:41:35 DEBUG } 20:41:35 DEBUG }, 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG ] 20:41:35 DEBUG "value": [ 20:41:35 DEBUG "484.14644957300317" 20:41:35 DEBUG } 20:41:35 DEBUG 1713728495, 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG "3447242752" 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 DEBUG ] 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 DEBUG } 20:41:35 INFO 20:41:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:41:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:35 INFO [loop_until]: OK (rc = 0) 20:41:35 DEBUG --- stdout --- 20:41:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22515m 8772Mi am-767f6b67b5-qcqz2 22138m 8982Mi am-767f6b67b5-vbltz 24048m 8862Mi ds-cts-0 20627m 11495Mi ds-cts-1 20109m 11630Mi ds-cts-2 20148m 11587Mi ds-idrepo-0 9615m 4026Mi ds-idrepo-1 9443m 4021Mi ds-idrepo-2 9909m 3947Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11991m 3198Mi 20:41:35 DEBUG --- stderr --- 20:41:35 DEBUG 20:41:35 INFO 20:41:35 INFO [loop_until]: kubectl --namespace=xlou top node 20:41:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:35 INFO [loop_until]: OK (rc = 0) 20:41:35 DEBUG --- stdout --- 20:41:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9911m 33% 7866Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21896m 73% 11588Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20569m 68% 14773Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5317Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22312m 74% 11895Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19750m 66% 3232Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24672m 82% 11175Mi 10% gke-xlou-cdm-ds-0f601751-j488 10305m 34% 7445Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20208m 67% 16955Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 9809m 32% 8064Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20747m 69% 14316Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11958m 75% 13474Mi 22% 20:41:35 DEBUG --- stderr --- 20:41:35 DEBUG 20:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728554" 20:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728554" 20:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728554" 20:42:34 INFO [http_cmd]: http status code OK 20:42:34 DEBUG --- status code --- 20:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:34 DEBUG --- http response --- 20:42:34 INFO [http_cmd]: http status code OK 20:42:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:42:34 DEBUG --- status code --- 20:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:34 DEBUG --- http response --- 20:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:34 DEBUG ----- output ----- 20:42:34 DEBUG { 20:42:34 DEBUG "status": "success", 20:42:34 DEBUG "data": { 20:42:34 DEBUG "resultType": "vector", 20:42:34 DEBUG "result": [ 20:42:34 DEBUG { 20:42:34 DEBUG "metric": { 20:42:34 DEBUG "instance": "10.142.0.107:9100" 20:42:34 DEBUG }, 20:42:34 DEBUG "value": [ 20:42:34 DEBUG 20:42:34 DEBUG 20:42:34 DEBUG 20:42:34 DEBUG [...] 20:42:34 DEBUG 20:42:34 DEBUG 20:42:34 DEBUG 20:42:34 DEBUG "instance": "10.142.0.43:9100" 20:42:34 DEBUG }, 20:42:34 DEBUG "value": [ 20:42:34 DEBUG 1713728554, 20:42:34 DEBUG "100357784.19999999" 20:42:34 DEBUG ] 20:42:34 DEBUG } 20:42:34 DEBUG ] 20:42:34 DEBUG } 20:42:34 DEBUG } 20:42:34 ERROR [http_cmd]: ERROR 20:42:34 ERROR --- status code --- 20:42:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:42:34 ERROR --- http response --- 20:42:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:42:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728555" 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "0" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG }, 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:42:35 DEBUG "pod": "ds-cts-2" 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG }, 20:42:35 DEBUG }, 20:42:35 DEBUG { 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "data": { 20:42:35 DEBUG "11.672176266540845" 20:42:35 DEBUG "0" 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG ] 20:42:35 DEBUG ] 20:42:35 DEBUG "result": [ 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG { 20:42:35 DEBUG ] 20:42:35 DEBUG ] 20:42:35 DEBUG "metric": { 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG "instance": "10.142.0.107:9100" 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "instance": "10.142.0.43:9100" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "0.17636666666666276" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728555" 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "instance": "10.142.0.107:9100" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "instance": "10.142.0.43:9100" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "100358829.7" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG { 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "data": { 20:42:35 DEBUG "data": { 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG "result": [ 20:42:35 DEBUG "result": [ 20:42:35 DEBUG "result": [ 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG { 20:42:35 DEBUG { 20:42:35 DEBUG { 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "pod": "ds-cts-0" 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:35 DEBUG }, 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG }, 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG { 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "data": { 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG "result": [ 20:42:35 DEBUG [...] 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG { 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "metric": { 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "ds-cts-2" 20:42:35 DEBUG }, 20:42:35 DEBUG "pod": "ds-idrepo-2" 20:42:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG }, 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "0" 20:42:35 DEBUG 20:42:35 DEBUG "0" 20:42:35 DEBUG "196.334" 20:42:35 DEBUG ] 20:42:35 DEBUG [...] 20:42:35 DEBUG ] 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG 20:42:35 DEBUG ] 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "3332.633333333333" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "instance": "10.142.0.107:9100" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "instance": "10.142.0.43:9100" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "0.05033333333334061" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "ds-idrepo-2" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "0" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "0" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728555" 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "instance": "10.142.0.107:9100" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "instance": "10.142.0.43:9100" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "4509454498816" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "ds-idrepo-2" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "0" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "data": { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "3314.9333333333334" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728555" 20:42:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728555" 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG --- http response --- 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG { 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG "data": { 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:35 DEBUG { 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG { 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG "metric": { 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG "data": { 20:42:35 DEBUG { 20:42:35 DEBUG "data": { 20:42:35 DEBUG { 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG }, 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "result": [ 20:42:35 DEBUG "data": { 20:42:35 DEBUG "result": [ 20:42:35 DEBUG "data": { 20:42:35 DEBUG "value": [ 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:42:35 DEBUG "data": { 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:35 DEBUG { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG { 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG 20:42:35 DEBUG ----- output ----- 20:42:35 INFO [http_cmd]: http status code OK 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "result": [ 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "result": [ 20:42:35 DEBUG 20:42:35 DEBUG { 20:42:35 DEBUG --- status code --- 20:42:35 DEBUG "result": [ 20:42:35 DEBUG { 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG { 20:42:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:35 DEBUG { 20:42:35 DEBUG 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:35 DEBUG { 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG }, 20:42:35 DEBUG "metric": { 20:42:35 DEBUG }, 20:42:35 DEBUG "metric": { 20:42:35 DEBUG [...] 20:42:35 DEBUG "data": { 20:42:35 DEBUG --- http response --- 20:42:35 DEBUG "metric": { 20:42:35 DEBUG "data": { 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "pod": "ds-cts-0" 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG "pod": "ds-cts-0" 20:42:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG 20:42:35 DEBUG }, 20:42:35 DEBUG 20:42:35 DEBUG }, 20:42:35 DEBUG 20:42:35 DEBUG "result": [ 20:42:35 DEBUG }, 20:42:35 DEBUG ----- output ----- 20:42:35 DEBUG "result": [ 20:42:35 DEBUG 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG { 20:42:35 DEBUG "value": [ 20:42:35 DEBUG { 20:42:35 DEBUG { 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:35 DEBUG "metric": { 20:42:35 DEBUG 20:42:35 DEBUG "status": "success", 20:42:35 DEBUG "metric": { 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG }, 20:42:35 DEBUG "pod": "ds-idrepo-0" 20:42:35 DEBUG 20:42:35 DEBUG "data": { 20:42:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "value": [ 20:42:35 DEBUG }, 20:42:35 DEBUG 20:42:35 DEBUG "resultType": "vector", 20:42:35 DEBUG }, 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 20:42:35 DEBUG [...] 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG [...] 20:42:35 DEBUG "result": [ 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "3330" 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG { 20:42:35 DEBUG 20:42:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:42:35 DEBUG 20:42:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:35 DEBUG 20:42:35 DEBUG ] 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "metric": { 20:42:35 DEBUG 20:42:35 DEBUG }, 20:42:35 DEBUG 20:42:35 DEBUG }, 20:42:35 DEBUG 20:42:35 DEBUG } 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:42:35 DEBUG 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "pod": "ds-cts-2" 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:42:35 DEBUG ] 20:42:35 DEBUG [...] 20:42:35 DEBUG "pod": "ds-cts-2" 20:42:35 DEBUG }, 20:42:35 DEBUG [...] 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG }, 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG }, 20:42:35 DEBUG } 20:42:35 DEBUG 20:42:35 DEBUG }, 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG "0" 20:42:35 DEBUG "value": [ 20:42:35 DEBUG "600258" 20:42:35 DEBUG "value": [ 20:42:35 DEBUG } 20:42:35 DEBUG 20:42:35 DEBUG "value": [ 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG ] 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG ] 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG 20:42:35 DEBUG 20:42:35 DEBUG } 20:42:35 DEBUG "0" 20:42:35 DEBUG } 20:42:35 DEBUG "0.000024071967003256814" 20:42:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:42:35 DEBUG "0.41523333333333323" 20:42:35 DEBUG 20:42:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:42:35 DEBUG ] 20:42:35 DEBUG ] 20:42:35 DEBUG ] 20:42:35 DEBUG ] 20:42:35 DEBUG }, 20:42:35 DEBUG ] 20:42:35 DEBUG [...] 20:42:35 DEBUG }, 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG "value": [ 20:42:35 DEBUG } 20:42:35 DEBUG 20:42:35 DEBUG "value": [ 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG ] 20:42:35 DEBUG 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG "0" 20:42:35 DEBUG } 20:42:35 DEBUG 20:42:35 DEBUG "3332.8" 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG }, 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG "value": [ 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG 1713728555, 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 DEBUG "3457601536" 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG ] 20:42:35 DEBUG } 20:42:35 DEBUG } 20:42:35 INFO 20:42:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:42:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:42:35 INFO 20:42:35 INFO [loop_until]: kubectl --namespace=xlou top node 20:42:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:42:35 INFO [loop_until]: OK (rc = 0) 20:42:35 DEBUG --- stdout --- 20:42:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21989m 8775Mi am-767f6b67b5-qcqz2 21819m 8987Mi am-767f6b67b5-vbltz 23945m 8872Mi ds-cts-0 20374m 11735Mi ds-cts-1 20238m 11834Mi ds-cts-2 20476m 11690Mi ds-idrepo-0 9460m 4027Mi ds-idrepo-1 9462m 4021Mi ds-idrepo-2 9645m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11276m 3219Mi 20:42:35 DEBUG --- stderr --- 20:42:35 DEBUG 20:42:35 INFO [loop_until]: OK (rc = 0) 20:42:35 DEBUG --- stdout --- 20:42:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6299Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9977m 33% 7864Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21882m 73% 11595Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20535m 68% 15019Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5291Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22290m 74% 11896Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19652m 65% 3233Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24807m 83% 11189Mi 10% gke-xlou-cdm-ds-0f601751-j488 10355m 34% 7439Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20202m 67% 16968Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10097m 33% 8064Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20819m 69% 14392Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11933m 75% 13488Mi 22% 20:42:35 DEBUG --- stderr --- 20:42:35 DEBUG 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 ERROR [http_cmd]: ERROR 20:43:35 ERROR --- status code --- 20:43:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:43:35 ERROR --- http response --- 20:43:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:43:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG "data": { 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "instance": "10.142.0.107:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "instance": "10.142.0.43:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "100711324.6" 20:43:35 DEBUG ] 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG ] 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG { 20:43:35 DEBUG } 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG } 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:43:35 DEBUG }, 20:43:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "11.68709498974793" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "ds-cts-2" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "instance": "10.142.0.107:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "instance": "10.142.0.43:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0.17553333333336052" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "instance": "10.142.0.107:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "instance": "10.142.0.43:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "100712512.46666667" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "ds-cts-0" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "ds-cts-2" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "3336.4" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "196.334" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG "metric": { 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG { 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG }, 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG "value": [ 20:43:35 DEBUG "data": { 20:43:35 DEBUG 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG "result": [ 20:43:35 DEBUG 20:43:35 DEBUG { 20:43:35 DEBUG { 20:43:35 DEBUG [...] 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "metric": { 20:43:35 DEBUG 20:43:35 DEBUG "data": { 20:43:35 DEBUG "instance": "10.142.0.107:9100" 20:43:35 DEBUG 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG }, 20:43:35 DEBUG 20:43:35 DEBUG "result": [ 20:43:35 DEBUG "value": [ 20:43:35 DEBUG "pod": "ds-idrepo-2" 20:43:35 DEBUG { 20:43:35 DEBUG 20:43:35 DEBUG }, 20:43:35 DEBUG "metric": { 20:43:35 DEBUG 20:43:35 DEBUG "value": [ 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG }, 20:43:35 DEBUG [...] 20:43:35 DEBUG "0" 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG ] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG } 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG ] 20:43:35 DEBUG 20:43:35 DEBUG "instance": "10.142.0.43:9100" 20:43:35 DEBUG } 20:43:35 DEBUG [...] 20:43:35 DEBUG }, 20:43:35 DEBUG } 20:43:35 DEBUG 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG 20:43:35 DEBUG "0.051666666666668935" 20:43:35 DEBUG "pod": "ds-idrepo-2" 20:43:35 DEBUG ] 20:43:35 DEBUG }, 20:43:35 DEBUG } 20:43:35 DEBUG "value": [ 20:43:35 DEBUG ] 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG } 20:43:35 DEBUG "0" 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "ds-idrepo-2" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "instance": "10.142.0.107:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "instance": "10.142.0.43:9100" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "4513629697024" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "3334.233333333333" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "3466166272" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "3338.633333333333" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0.000035475363437556934" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728615" 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "3336.733333333333" 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG { 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "data": { 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "result": [ 20:43:35 DEBUG { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:43:35 DEBUG }, 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:35 DEBUG "value": [ 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG 20:43:35 DEBUG { 20:43:35 DEBUG 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG 20:43:35 DEBUG "data": { 20:43:35 DEBUG [...] 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG 20:43:35 DEBUG "result": [ 20:43:35 DEBUG 20:43:35 DEBUG { 20:43:35 DEBUG 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:43:35 DEBUG "pod": "ds-cts-0" 20:43:35 DEBUG }, 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG 20:43:35 DEBUG "600099" 20:43:35 DEBUG 20:43:35 DEBUG ] 20:43:35 DEBUG 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG } 20:43:35 INFO [http_cmd]: http status code OK 20:43:35 DEBUG [...] 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG ] 20:43:35 DEBUG --- status code --- 20:43:35 DEBUG 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG } 20:43:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:35 DEBUG 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG } 20:43:35 DEBUG --- http response --- 20:43:35 DEBUG 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:35 DEBUG "pod": "ds-cts-2" 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG ----- output ----- 20:43:35 DEBUG }, 20:43:35 DEBUG { 20:43:35 DEBUG { 20:43:35 DEBUG "value": [ 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG "status": "success", 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "data": { 20:43:35 DEBUG "data": { 20:43:35 DEBUG "0.47776666666666656" 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG "resultType": "vector", 20:43:35 DEBUG ] 20:43:35 DEBUG "result": [ 20:43:35 DEBUG "result": [ 20:43:35 DEBUG } 20:43:35 DEBUG { 20:43:35 DEBUG { 20:43:35 DEBUG ] 20:43:35 DEBUG "metric": { 20:43:35 DEBUG "metric": { 20:43:35 DEBUG } 20:43:35 DEBUG "pod": "ds-idrepo-0" 20:43:35 DEBUG "pod": "ds-cts-0" 20:43:35 DEBUG } 20:43:35 DEBUG }, 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG [...] 20:43:35 DEBUG [...] 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG 20:43:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:43:35 DEBUG "pod": "ds-cts-2" 20:43:35 DEBUG }, 20:43:35 DEBUG }, 20:43:35 DEBUG "value": [ 20:43:35 DEBUG "value": [ 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG 1713728615, 20:43:35 DEBUG "0" 20:43:35 DEBUG "0" 20:43:35 DEBUG ] 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 DEBUG ] 20:43:35 DEBUG ] 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 DEBUG } 20:43:35 INFO 20:43:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:43:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:43:35 INFO 20:43:35 INFO [loop_until]: kubectl --namespace=xlou top node 20:43:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:43:35 INFO [loop_until]: OK (rc = 0) 20:43:35 DEBUG --- stdout --- 20:43:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22374m 8778Mi am-767f6b67b5-qcqz2 21833m 8989Mi am-767f6b67b5-vbltz 23739m 8867Mi ds-cts-0 20487m 11848Mi ds-cts-1 20568m 11856Mi ds-cts-2 20656m 11753Mi ds-idrepo-0 9548m 4027Mi ds-idrepo-1 9271m 4021Mi ds-idrepo-2 9779m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11534m 3212Mi 20:43:35 DEBUG --- stderr --- 20:43:35 DEBUG 20:43:35 INFO [loop_until]: OK (rc = 0) 20:43:35 DEBUG --- stdout --- 20:43:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10021m 33% 7867Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21982m 73% 11599Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3363Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19704m 65% 15151Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5307Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22466m 75% 11892Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19772m 66% 3228Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24743m 82% 11180Mi 10% gke-xlou-cdm-ds-0f601751-j488 10355m 34% 7440Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20132m 67% 16988Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10052m 33% 8069Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20577m 68% 14471Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11923m 75% 13483Mi 22% 20:43:35 DEBUG --- stderr --- 20:43:35 DEBUG 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 ERROR [http_cmd]: ERROR 20:44:35 ERROR --- status code --- 20:44:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:44:35 ERROR --- http response --- 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:44:35 DEBUG --- status code --- 20:44:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "instance": "10.142.0.107:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "instance": "10.142.0.43:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "100665479.1" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-idrepo-2" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "17706.8749128434" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "instance": "10.142.0.107:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "instance": "10.142.0.43:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "0.25903333333328077" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-cts-2" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "0" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "11.673975212359746" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "instance": "10.142.0.107:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "instance": "10.142.0.43:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "100666448.6" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "3344.6666666666665" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-cts-0" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-cts-2" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "0" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:35 DEBUG }, 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:35 DEBUG }, 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "196.334" 20:44:35 DEBUG "0" 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG }, 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-idrepo-2" 20:44:35 DEBUG "pod": "ds-idrepo-2" 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:35 DEBUG }, 20:44:35 DEBUG }, 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG "value": [ 20:44:35 DEBUG "value": [ 20:44:35 DEBUG { 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "0" 20:44:35 DEBUG "0" 20:44:35 DEBUG "data": { 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG "result": [ 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG { 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG "metric": { 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG "instance": "10.142.0.107:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "instance": "10.142.0.43:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "0.06966666666666584" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-idrepo-2" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "0" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG ] 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG } 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG ] 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "3331.0666666666666" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "instance": "10.142.0.107:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "instance": "10.142.0.43:9100" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "4518330622976" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG --- status code --- 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG { 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:44:35 DEBUG "data": { 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "data": { 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG }, 20:44:35 DEBUG "result": [ 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "value": [ 20:44:35 DEBUG { 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-cts-0" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG }, 20:44:35 DEBUG 20:44:35 DEBUG }, 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG }, 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG "value": [ 20:44:35 DEBUG [...] 20:44:35 DEBUG "value": [ 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG "value": [ 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG [...] 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:35 DEBUG [...] 20:44:35 DEBUG [...] 20:44:35 DEBUG { 20:44:35 DEBUG 20:44:35 DEBUG { 20:44:35 DEBUG }, 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "data": { 20:44:35 DEBUG 20:44:35 DEBUG "data": { 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "pod": "ds-idrepo-2" 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "600105" 20:44:35 DEBUG "pod": "ds-cts-2" 20:44:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:44:35 DEBUG "result": [ 20:44:35 DEBUG }, 20:44:35 DEBUG "result": [ 20:44:35 DEBUG ] 20:44:35 DEBUG }, 20:44:35 DEBUG }, 20:44:35 DEBUG { 20:44:35 DEBUG "value": [ 20:44:35 DEBUG { 20:44:35 DEBUG } 20:44:35 DEBUG "value": [ 20:44:35 DEBUG "value": [ 20:44:35 DEBUG "metric": { 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "metric": { 20:44:35 DEBUG ] 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:35 DEBUG "9.724986996234746" 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG } 20:44:35 DEBUG "0" 20:44:35 DEBUG "3470938112" 20:44:35 DEBUG }, 20:44:35 DEBUG ] 20:44:35 DEBUG }, 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG "value": [ 20:44:35 DEBUG } 20:44:35 DEBUG "value": [ 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG 20:44:35 DEBUG ] 20:44:35 DEBUG 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG 20:44:35 DEBUG } 20:44:35 DEBUG 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG 20:44:35 DEBUG } 20:44:35 DEBUG 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG [...] 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:35 DEBUG "pod": "ds-idrepo-2" 20:44:35 DEBUG }, 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "3332.733333333333" 20:44:35 DEBUG "0" 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "3336.7666666666664" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728675" 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 INFO [http_cmd]: http status code OK 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG --- status code --- 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG --- http response --- 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-cts-0" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-cts-2" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "0.5698000000000003" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:44:35 DEBUG ----- output ----- 20:44:35 DEBUG { 20:44:35 DEBUG "status": "success", 20:44:35 DEBUG "data": { 20:44:35 DEBUG "resultType": "vector", 20:44:35 DEBUG "result": [ 20:44:35 DEBUG { 20:44:35 DEBUG "metric": { 20:44:35 DEBUG "pod": "ds-idrepo-0" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG [...] 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG 20:44:35 DEBUG "pod": "ds-idrepo-2" 20:44:35 DEBUG }, 20:44:35 DEBUG "value": [ 20:44:35 DEBUG 1713728675, 20:44:35 DEBUG "5426.300376516525" 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG ] 20:44:35 DEBUG } 20:44:35 DEBUG } 20:44:35 INFO 20:44:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:44:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:44:35 INFO 20:44:35 INFO [loop_until]: kubectl --namespace=xlou top node 20:44:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:44:35 INFO [loop_until]: OK (rc = 0) 20:44:35 DEBUG --- stdout --- 20:44:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22301m 8778Mi am-767f6b67b5-qcqz2 22251m 8987Mi am-767f6b67b5-vbltz 24219m 8865Mi ds-cts-0 20682m 11841Mi ds-cts-1 19862m 11904Mi ds-cts-2 20252m 11742Mi ds-idrepo-0 9762m 4026Mi ds-idrepo-1 9495m 4021Mi ds-idrepo-2 9869m 3947Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11361m 3218Mi 20:44:35 DEBUG --- stderr --- 20:44:35 DEBUG 20:44:35 INFO [loop_until]: OK (rc = 0) 20:44:35 DEBUG --- stdout --- 20:44:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9974m 33% 7863Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22003m 73% 11597Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3370Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20072m 67% 15141Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 5296Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 21590m 72% 11885Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19759m 66% 3214Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24644m 82% 11181Mi 10% gke-xlou-cdm-ds-0f601751-j488 10372m 34% 7442Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19651m 65% 17028Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10088m 33% 8068Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20326m 68% 14451Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11993m 75% 13482Mi 22% 20:44:35 DEBUG --- stderr --- 20:44:35 DEBUG 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "instance": "10.142.0.107:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "instance": "10.142.0.43:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "100245869.33333333" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 ERROR [http_cmd]: ERROR 20:45:35 ERROR --- status code --- 20:45:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:45:35 ERROR --- http response --- 20:45:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:45:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "ds-cts-2" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "instance": "10.142.0.107:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "instance": "10.142.0.43:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0.4116999999999848" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "11.68756427028108" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "instance": "10.142.0.107:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "instance": "10.142.0.43:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "100246660.5" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "3316.5" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-cts-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "ds-cts-2" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG "0" 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG ] 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "instance": "10.142.0.107:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "instance": "10.142.0.43:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0.07766666666665857" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "ds-idrepo-2" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "196.334" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "ds-idrepo-2" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "3327.5666666666666" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "ds-idrepo-2" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "instance": "10.142.0.107:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "instance": "10.142.0.43:9100" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "4523550620672" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 DEBUG "data": { 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728735" 20:45:35 DEBUG "resultType": "vector", 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 DEBUG "result": [ 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG { 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG "metric": { 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG "pod": "ds-cts-0" 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "ds-cts-2" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0.5384" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "599697.5" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-cts-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "ds-cts-2" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "3472977920" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "3335.5333333333333" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728735" 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "3319.6" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO [http_cmd]: http status code OK 20:45:35 DEBUG --- status code --- 20:45:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:35 DEBUG --- http response --- 20:45:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:45:35 DEBUG ----- output ----- 20:45:35 DEBUG { 20:45:35 DEBUG "status": "success", 20:45:35 DEBUG "data": { 20:45:35 DEBUG "resultType": "vector", 20:45:35 DEBUG "result": [ 20:45:35 DEBUG { 20:45:35 DEBUG "metric": { 20:45:35 DEBUG "pod": "ds-idrepo-0" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG [...] 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG 20:45:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:45:35 DEBUG }, 20:45:35 DEBUG "value": [ 20:45:35 DEBUG 1713728735, 20:45:35 DEBUG "0" 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG ] 20:45:35 DEBUG } 20:45:35 DEBUG } 20:45:35 INFO 20:45:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:45:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:45:35 INFO 20:45:35 INFO [loop_until]: kubectl --namespace=xlou top node 20:45:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:45:35 INFO [loop_until]: OK (rc = 0) 20:45:35 DEBUG --- stdout --- 20:45:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21724m 8776Mi am-767f6b67b5-qcqz2 21994m 8989Mi am-767f6b67b5-vbltz 24950m 8873Mi ds-cts-0 20024m 11841Mi ds-cts-1 20187m 11904Mi ds-cts-2 20360m 11767Mi ds-idrepo-0 9484m 4027Mi ds-idrepo-1 9485m 3973Mi ds-idrepo-2 9566m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11492m 3218Mi 20:45:35 DEBUG --- stderr --- 20:45:35 DEBUG 20:45:35 INFO [loop_until]: OK (rc = 0) 20:45:35 DEBUG --- stdout --- 20:45:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6300Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10009m 33% 7867Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21880m 73% 11592Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3368Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20125m 67% 15147Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 144m 0% 5305Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22195m 74% 11903Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19742m 66% 3238Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24792m 83% 11170Mi 10% gke-xlou-cdm-ds-0f601751-j488 10340m 34% 7442Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19908m 66% 17023Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10159m 34% 8042Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20327m 68% 14457Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11877m 74% 13483Mi 22% 20:45:35 DEBUG --- stderr --- 20:45:35 DEBUG 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728795" 20:46:35 ERROR [http_cmd]: ERROR 20:46:35 ERROR --- status code --- 20:46:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:46:35 ERROR --- http response --- 20:46:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:46:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-1" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG ] 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "instance": "10.142.0.107:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "instance": "10.142.0.43:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "99926940.26666667" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-0" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "ds-cts-2" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "instance": "10.142.0.107:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "instance": "10.142.0.43:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0.4073333333333911" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "11.64454625263491" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "instance": "10.142.0.107:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "instance": "10.142.0.43:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "99929776.63333333" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "3326.866666666667" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "196.335" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-cts-0" 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-0" 20:46:35 DEBUG }, 20:46:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "ds-cts-2" 20:46:35 DEBUG 20:46:35 DEBUG "pod": "ds-idrepo-2" 20:46:35 DEBUG }, 20:46:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG ] 20:46:35 DEBUG "0" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "instance": "10.142.0.107:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "instance": "10.142.0.43:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0.08166666666666438" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-0" 20:46:35 DEBUG "instance": "10.142.0.107:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "ds-idrepo-2" 20:46:35 DEBUG "instance": "10.142.0.43:9100" 20:46:35 DEBUG }, 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG "4528811783168" 20:46:35 DEBUG ] 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-0" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "ds-idrepo-2" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "3335.7999999999997" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "3476979712" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-1" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-1" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0.000018250060259614864" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG { 20:46:35 DEBUG { 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "data": { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-cts-0" 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:35 DEBUG }, 20:46:35 DEBUG "result": [ 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG { 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG "metric": { 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "ds-cts-0" 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG }, 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG "value": [ 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:35 DEBUG [...] 20:46:35 DEBUG "pod": "ds-cts-2" 20:46:35 DEBUG }, 20:46:35 DEBUG 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "600165.5" 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG "pod": "ds-cts-2" 20:46:35 DEBUG "0.5174000000000006" 20:46:35 DEBUG ] 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG }, 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG "value": [ 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG "0" 20:46:35 DEBUG { 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 DEBUG "data": { 20:46:35 DEBUG ] 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG } 20:46:35 DEBUG "result": [ 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 DEBUG } 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "3329.0333333333333" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728795" 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "3330.633333333333" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO [http_cmd]: http status code OK 20:46:35 DEBUG --- status code --- 20:46:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:35 DEBUG --- http response --- 20:46:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:46:35 DEBUG ----- output ----- 20:46:35 DEBUG { 20:46:35 DEBUG "status": "success", 20:46:35 DEBUG "data": { 20:46:35 DEBUG "resultType": "vector", 20:46:35 DEBUG "result": [ 20:46:35 DEBUG { 20:46:35 DEBUG "metric": { 20:46:35 DEBUG "pod": "ds-idrepo-1" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG [...] 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG 20:46:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:46:35 DEBUG }, 20:46:35 DEBUG "value": [ 20:46:35 DEBUG 1713728795, 20:46:35 DEBUG "0" 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG ] 20:46:35 DEBUG } 20:46:35 DEBUG } 20:46:35 INFO 20:46:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:46:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:46:35 INFO [loop_until]: OK (rc = 0) 20:46:35 DEBUG --- stdout --- 20:46:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22540m 8777Mi am-767f6b67b5-qcqz2 22118m 8988Mi am-767f6b67b5-vbltz 24337m 8862Mi ds-cts-0 19969m 11970Mi ds-cts-1 19868m 11930Mi ds-cts-2 20649m 11821Mi ds-idrepo-0 9430m 4026Mi ds-idrepo-1 9646m 3985Mi ds-idrepo-2 9575m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11427m 3220Mi 20:46:35 DEBUG --- stderr --- 20:46:35 DEBUG 20:46:35 INFO 20:46:35 INFO [loop_until]: kubectl --namespace=xlou top node 20:46:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:46:35 INFO [loop_until]: OK (rc = 0) 20:46:35 DEBUG --- stdout --- 20:46:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6297Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9980m 33% 7859Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21218m 71% 11603Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3359Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20194m 67% 15277Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5293Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22359m 74% 11892Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19714m 66% 3234Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24661m 82% 11177Mi 10% gke-xlou-cdm-ds-0f601751-j488 10365m 34% 7439Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19863m 66% 17048Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10106m 33% 8029Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20510m 68% 14492Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11875m 74% 13483Mi 22% 20:46:35 DEBUG --- stderr --- 20:46:35 DEBUG 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728855" 20:47:35 ERROR [http_cmd]: ERROR 20:47:35 ERROR --- status code --- 20:47:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:47:35 ERROR --- http response --- 20:47:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:47:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "instance": "10.142.0.107:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "instance": "10.142.0.43:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "100364528.26666667" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "ds-idrepo-1" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "ds-idrepo-2" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "147975963.04978055" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "ds-idrepo-0" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "ds-cts-2" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "0" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "11.640697116739311" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "instance": "10.142.0.107:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "instance": "10.142.0.43:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "0.4099666666667095" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "instance": "10.142.0.107:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "instance": "10.142.0.43:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "100365429.73333332" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:35 DEBUG "pod": "ds-cts-0" 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:35 DEBUG "pod": "ds-cts-2" 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "0" 20:47:35 DEBUG "0" 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "3329.211385907606" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "196.33550000000002" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "ds-idrepo-0" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "ds-idrepo-2" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "0" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "instance": "10.142.0.107:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "instance": "10.142.0.43:9100" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "0.07799999999999727" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "3358.976068262116" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "instance": "10.142.0.107:9100" 20:47:35 DEBUG "pod": "ds-idrepo-0" 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "instance": "10.142.0.43:9100" 20:47:35 DEBUG "pod": "ds-idrepo-2" 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "4534012816384" 20:47:35 DEBUG "0" 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "ds-idrepo-0" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "ds-idrepo-2" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "0" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "3476033536" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "ds-idrepo-1" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "ds-idrepo-2" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "0" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "ds-idrepo-1" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "ds-idrepo-2" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "2221.6667166256293" 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728855" 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "metric": { 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 INFO [http_cmd]: http status code OK 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:35 DEBUG "pod": "ds-idrepo-1" 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG --- status code --- 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG --- http response --- 20:47:35 DEBUG 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:35 DEBUG 20:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG 20:47:35 DEBUG ----- output ----- 20:47:35 DEBUG { 20:47:35 DEBUG 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 DEBUG [...] 20:47:35 DEBUG { 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG [...] 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG 20:47:35 DEBUG "status": "success", 20:47:35 DEBUG "data": { 20:47:35 DEBUG 20:47:35 DEBUG "data": { 20:47:35 DEBUG "data": { 20:47:35 DEBUG 20:47:35 DEBUG "data": { 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG 20:47:35 DEBUG "resultType": "vector", 20:47:35 DEBUG "result": [ 20:47:35 DEBUG 20:47:35 DEBUG "result": [ 20:47:35 DEBUG "result": [ 20:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:35 DEBUG "result": [ 20:47:35 DEBUG { 20:47:35 DEBUG "pod": "ds-idrepo-2" 20:47:35 DEBUG { 20:47:35 DEBUG { 20:47:35 DEBUG }, 20:47:35 DEBUG { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG }, 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "metric": { 20:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:47:35 DEBUG "pod": "ds-cts-0" 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "pod": "ds-cts-0" 20:47:35 DEBUG }, 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG "3333.1444570361973" 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "5.448157418519342" 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG ] 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 20:47:35 DEBUG ] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG } 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG } 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG ] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG ] 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG } 20:47:35 DEBUG 20:47:35 DEBUG [...] 20:47:35 DEBUG } 20:47:35 DEBUG [...] 20:47:35 DEBUG [...] 20:47:35 DEBUG } 20:47:35 DEBUG [...] 20:47:35 DEBUG 20:47:35 DEBUG } 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG 20:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:47:35 DEBUG "pod": "ds-cts-2" 20:47:35 DEBUG "pod": "ds-cts-2" 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG }, 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG "value": [ 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG 1713728855, 20:47:35 DEBUG "3333.6444237050864" 20:47:35 DEBUG "600048" 20:47:35 DEBUG "0" 20:47:35 DEBUG "0.5440666666666663" 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG ] 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 DEBUG } 20:47:35 INFO 20:47:35 INFO [loop_until]: kubectl --namespace=xlou top pods 20:47:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:47:35 INFO [loop_until]: OK (rc = 0) 20:47:35 DEBUG --- stdout --- 20:47:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21955m 8777Mi am-767f6b67b5-qcqz2 22816m 8988Mi am-767f6b67b5-vbltz 24895m 8865Mi ds-cts-0 19954m 11976Mi ds-cts-1 20129m 11934Mi ds-cts-2 20321m 11814Mi ds-idrepo-0 9537m 4026Mi ds-idrepo-1 9371m 3985Mi ds-idrepo-2 9628m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11671m 3225Mi 20:47:35 DEBUG --- stderr --- 20:47:35 DEBUG 20:47:36 INFO 20:47:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:47:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:47:36 INFO [loop_until]: OK (rc = 0) 20:47:36 DEBUG --- stdout --- 20:47:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6296Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9666m 32% 7870Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21899m 73% 11592Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20026m 67% 15275Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5300Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22192m 74% 11904Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19142m 64% 3232Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24691m 82% 11178Mi 10% gke-xlou-cdm-ds-0f601751-j488 10364m 34% 7440Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20039m 67% 17056Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10104m 33% 8028Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20355m 68% 14530Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11874m 74% 13480Mi 22% 20:47:36 DEBUG --- stderr --- 20:47:36 DEBUG 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 ERROR [http_cmd]: ERROR 20:48:35 ERROR --- status code --- 20:48:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:48:35 ERROR --- http response --- 20:48:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:48:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "11.699786905068429" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "instance": "10.142.0.107:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "instance": "10.142.0.43:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "100567348.46666667" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "ds-cts-2" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "instance": "10.142.0.107:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "instance": "10.142.0.43:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0.33423333333334043" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "instance": "10.142.0.107:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "instance": "10.142.0.43:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "100559828.23333333" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "instance": "10.142.0.107:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "instance": "10.142.0.43:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0.05266666666666803" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG { 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG "data": { 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "pod": "ds-cts-0" 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG }, 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "value": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:35 DEBUG }, 20:48:35 DEBUG 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG [...] 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG "pod": "ds-cts-2" 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG }, 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:35 DEBUG "0" 20:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:35 DEBUG }, 20:48:35 DEBUG ] 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG } 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG ] 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0" 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG } 20:48:35 DEBUG "196.336" 20:48:35 DEBUG ] 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG { 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG } 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "3312.004001333778" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "ds-idrepo-2" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "3305.568522840947" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG { 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG "data": { 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG }, 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "value": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG "instance": "10.142.0.107:9100" 20:48:35 DEBUG }, 20:48:35 DEBUG 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG [...] 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG { 20:48:35 DEBUG [...] 20:48:35 DEBUG "pod": "ds-idrepo-2" 20:48:35 DEBUG [...] 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG 20:48:35 DEBUG }, 20:48:35 DEBUG 20:48:35 DEBUG "data": { 20:48:35 DEBUG 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG 20:48:35 DEBUG "result": [ 20:48:35 DEBUG "pod": "ds-idrepo-2" 20:48:35 DEBUG "0" 20:48:35 DEBUG "instance": "10.142.0.43:9100" 20:48:35 DEBUG { 20:48:35 DEBUG }, 20:48:35 DEBUG ] 20:48:35 DEBUG }, 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "value": [ 20:48:35 DEBUG } 20:48:35 DEBUG "value": [ 20:48:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG ] 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG }, 20:48:35 DEBUG "0" 20:48:35 DEBUG } 20:48:35 DEBUG "4538866076672" 20:48:35 DEBUG "value": [ 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 DEBUG 20:48:35 DEBUG ] 20:48:35 DEBUG ] 20:48:35 DEBUG 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 DEBUG [...] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "3479363584" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "3312.804268089363" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728915" 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "data": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "result": [ 20:48:35 DEBUG { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "pod": "ds-idrepo-0" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:48:35 DEBUG }, 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "0" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG --- http response --- 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG { 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:35 DEBUG "data": { 20:48:35 INFO [http_cmd]: http status code OK 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG --- status code --- 20:48:35 DEBUG { 20:48:35 DEBUG { 20:48:35 DEBUG "result": [ 20:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG { 20:48:35 DEBUG --- http response --- 20:48:35 DEBUG "data": { 20:48:35 DEBUG "data": { 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:35 DEBUG "result": [ 20:48:35 DEBUG ----- output ----- 20:48:35 DEBUG "result": [ 20:48:35 DEBUG }, 20:48:35 DEBUG { 20:48:35 DEBUG { 20:48:35 DEBUG { 20:48:35 DEBUG "value": [ 20:48:35 DEBUG "metric": { 20:48:35 DEBUG "status": "success", 20:48:35 DEBUG "metric": { 20:48:35 DEBUG 20:48:35 DEBUG "pod": "ds-cts-0" 20:48:35 DEBUG "data": { 20:48:35 DEBUG "pod": "ds-cts-0" 20:48:35 DEBUG 20:48:35 DEBUG }, 20:48:35 DEBUG "resultType": "vector", 20:48:35 DEBUG }, 20:48:35 DEBUG 20:48:35 DEBUG "value": [ 20:48:35 DEBUG "result": [ 20:48:35 DEBUG "value": [ 20:48:35 DEBUG [...] 20:48:35 DEBUG 20:48:35 DEBUG { 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "metric": { 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG [...] 20:48:35 DEBUG }, 20:48:35 DEBUG [...] 20:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:35 DEBUG 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG }, 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG "pod": "ds-cts-2" 20:48:35 DEBUG 20:48:35 DEBUG "pod": "ds-cts-2" 20:48:35 DEBUG "3338.6795598532844" 20:48:35 DEBUG }, 20:48:35 DEBUG [...] 20:48:35 DEBUG }, 20:48:35 DEBUG ] 20:48:35 DEBUG "value": [ 20:48:35 DEBUG 20:48:35 DEBUG "value": [ 20:48:35 DEBUG } 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG ] 20:48:35 DEBUG "0" 20:48:35 DEBUG 20:48:35 DEBUG "0.4762" 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 DEBUG }, 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG "value": [ 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG 1713728915, 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:35 DEBUG "600128" 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG ] 20:48:35 DEBUG } 20:48:35 DEBUG } 20:48:36 INFO 20:48:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:48:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:48:36 INFO [loop_until]: OK (rc = 0) 20:48:36 DEBUG --- stdout --- 20:48:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22442m 8777Mi am-767f6b67b5-qcqz2 21363m 8990Mi am-767f6b67b5-vbltz 24424m 8886Mi ds-cts-0 20242m 11968Mi ds-cts-1 19908m 12061Mi ds-cts-2 20798m 11885Mi ds-idrepo-0 9926m 4026Mi ds-idrepo-1 9305m 3985Mi ds-idrepo-2 9642m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11665m 3299Mi 20:48:36 DEBUG --- stderr --- 20:48:36 DEBUG 20:48:36 INFO 20:48:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:48:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:48:36 INFO [loop_until]: OK (rc = 0) 20:48:36 DEBUG --- stdout --- 20:48:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10007m 33% 7875Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21946m 73% 11593Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3352Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20333m 68% 15275Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 5307Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22304m 74% 11893Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19711m 66% 3227Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24859m 83% 11197Mi 10% gke-xlou-cdm-ds-0f601751-j488 10054m 33% 7431Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20037m 67% 17158Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10157m 34% 8044Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20567m 68% 14594Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11885m 74% 13549Mi 23% 20:48:36 DEBUG --- stderr --- 20:48:36 DEBUG 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 ERROR [http_cmd]: ERROR 20:49:35 ERROR --- status code --- 20:49:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:49:35 ERROR --- http response --- 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG { 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "data": { 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG "result": [ 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG { 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "instance": "10.142.0.107:9100" 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:49:35 DEBUG }, 20:49:35 DEBUG }, 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG "value": [ 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG [...] 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "instance": "10.142.0.43:9100" 20:49:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:49:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:49:35 DEBUG }, 20:49:35 DEBUG }, 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG "value": [ 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "100297711.56666666" 20:49:35 DEBUG "137.78761439288803" 20:49:35 DEBUG "11.622324055460838" 20:49:35 DEBUG ] 20:49:35 DEBUG ] 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG ] 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "ds-cts-2" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "instance": "10.142.0.107:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "instance": "10.142.0.43:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0.4021333333333359" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "instance": "10.142.0.107:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "instance": "10.142.0.43:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "100298660.33333334" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-cts-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "ds-cts-2" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "instance": "10.142.0.107:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "instance": "10.142.0.43:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0.08133333333332946" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "196.336" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "3327.9666666666667" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:35 DEBUG }, 20:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "ds-idrepo-2" 20:49:35 DEBUG 20:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:35 DEBUG }, 20:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0" 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "3333.5333333333333" 20:49:35 DEBUG ] 20:49:35 DEBUG "0" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "instance": "10.142.0.107:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "instance": "10.142.0.43:9100" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "4543895776256" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "ds-idrepo-2" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "ds-idrepo-2" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "3558514688" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "1515.6637583217685" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG { 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "3335.233333333333" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0.0018455668672888487" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-cts-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "ds-cts-2" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-idrepo-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713728975" 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "3333.3333333333335" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG --- http response --- 20:49:35 INFO [http_cmd]: http status code OK 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:35 DEBUG --- status code --- 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:35 DEBUG { 20:49:35 DEBUG --- http response --- 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "600104.5" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:35 DEBUG ----- output ----- 20:49:35 DEBUG { 20:49:35 DEBUG "status": "success", 20:49:35 DEBUG "data": { 20:49:35 DEBUG "resultType": "vector", 20:49:35 DEBUG "result": [ 20:49:35 DEBUG { 20:49:35 DEBUG "metric": { 20:49:35 DEBUG "pod": "ds-cts-0" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG [...] 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG 20:49:35 DEBUG "pod": "ds-cts-2" 20:49:35 DEBUG }, 20:49:35 DEBUG "value": [ 20:49:35 DEBUG 1713728975, 20:49:35 DEBUG "0.5437333333333323" 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG ] 20:49:35 DEBUG } 20:49:35 DEBUG } 20:49:36 INFO 20:49:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:49:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:49:36 INFO [loop_until]: OK (rc = 0) 20:49:36 DEBUG --- stdout --- 20:49:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22012m 8779Mi am-767f6b67b5-qcqz2 22761m 8997Mi am-767f6b67b5-vbltz 24517m 8875Mi ds-cts-0 19525m 11966Mi ds-cts-1 19918m 12064Mi ds-cts-2 20311m 11886Mi ds-idrepo-0 9640m 4026Mi ds-idrepo-1 9688m 3998Mi ds-idrepo-2 9659m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1180Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11967m 3302Mi 20:49:36 DEBUG --- stderr --- 20:49:36 DEBUG 20:49:36 INFO 20:49:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:49:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:49:36 INFO [loop_until]: OK (rc = 0) 20:49:36 DEBUG --- stdout --- 20:49:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6296Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10008m 33% 7852Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21924m 73% 11589Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19700m 65% 15285Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5302Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22431m 75% 11906Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19781m 66% 3224Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24682m 82% 11176Mi 10% gke-xlou-cdm-ds-0f601751-j488 10369m 34% 7449Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19819m 66% 17164Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10154m 34% 8049Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20312m 68% 14594Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11878m 74% 13553Mi 23% 20:49:36 DEBUG --- stderr --- 20:49:36 DEBUG 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "instance": "10.142.0.107:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "instance": "10.142.0.43:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "100098619.53333333" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729035" 20:50:35 ERROR [http_cmd]: ERROR 20:50:35 ERROR --- status code --- 20:50:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:50:35 ERROR --- http response --- 20:50:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:50:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG "result": [ 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG { 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG "metric": { 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:35 DEBUG 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:50:35 DEBUG { 20:50:35 DEBUG }, 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "value": [ 20:50:35 DEBUG "data": { 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "98071111.3530595" 20:50:35 DEBUG "result": [ 20:50:35 DEBUG ] 20:50:35 DEBUG { 20:50:35 DEBUG } 20:50:35 DEBUG "metric": { 20:50:35 DEBUG ] 20:50:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:50:35 DEBUG } 20:50:35 DEBUG }, 20:50:35 DEBUG } 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "11.619672007436943" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "ds-cts-2" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "instance": "10.142.0.107:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "instance": "10.142.0.43:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0.4074000000000789" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "instance": "10.142.0.107:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "instance": "10.142.0.43:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "100099439.66666666" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "196.336" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-cts-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "ds-cts-2" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "8.833333333333334" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "3338.7" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "instance": "10.142.0.107:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "instance": "10.142.0.43:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0.08199999999999741" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:35 DEBUG }, 20:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "ds-idrepo-2" 20:50:35 DEBUG 20:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:35 DEBUG }, 20:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0" 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "3349.1" 20:50:35 DEBUG ] 20:50:35 DEBUG "0" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "3562291200" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "ds-idrepo-2" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "ds-idrepo-2" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "instance": "10.142.0.107:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "instance": "10.142.0.43:9100" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "4549209164800" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-cts-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "ds-cts-2" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "3346.0666666666666" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "23.16853739178729" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-idrepo-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729035" 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "ds-cts-0" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "ds-cts-2" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "0.48143333333333277" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 INFO [http_cmd]: http status code OK 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG --- status code --- 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG --- http response --- 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "599950.5" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:35 DEBUG ----- output ----- 20:50:35 DEBUG { 20:50:35 DEBUG "status": "success", 20:50:35 DEBUG "data": { 20:50:35 DEBUG "resultType": "vector", 20:50:35 DEBUG "result": [ 20:50:35 DEBUG { 20:50:35 DEBUG "metric": { 20:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG [...] 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG 20:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:50:35 DEBUG }, 20:50:35 DEBUG "value": [ 20:50:35 DEBUG 1713729035, 20:50:35 DEBUG "3326.9666666666667" 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG ] 20:50:35 DEBUG } 20:50:35 DEBUG } 20:50:36 INFO 20:50:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:50:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:50:36 INFO [loop_until]: OK (rc = 0) 20:50:36 DEBUG --- stdout --- 20:50:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22030m 8800Mi am-767f6b67b5-qcqz2 22573m 8992Mi am-767f6b67b5-vbltz 24136m 8880Mi ds-cts-0 19922m 12029Mi ds-cts-1 19821m 12107Mi ds-cts-2 20671m 11941Mi ds-idrepo-0 9360m 4026Mi ds-idrepo-1 9437m 3998Mi ds-idrepo-2 9709m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11500m 3305Mi 20:50:36 DEBUG --- stderr --- 20:50:36 DEBUG 20:50:36 INFO 20:50:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:50:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:50:36 INFO [loop_until]: OK (rc = 0) 20:50:36 DEBUG --- stdout --- 20:50:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6302Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9971m 33% 7864Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22028m 73% 11630Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19748m 66% 15342Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5321Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22315m 74% 11910Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19683m 65% 3230Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24687m 82% 11177Mi 10% gke-xlou-cdm-ds-0f601751-j488 10333m 34% 7447Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19996m 66% 17218Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10144m 33% 8035Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20637m 69% 14653Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11862m 74% 13556Mi 23% 20:50:36 DEBUG --- stderr --- 20:50:36 DEBUG 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "instance": "10.142.0.107:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "instance": "10.142.0.43:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "100486210.46666667" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729095" 20:51:35 ERROR [http_cmd]: ERROR 20:51:35 ERROR --- status code --- 20:51:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:51:35 ERROR --- http response --- 20:51:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:51:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "idm-b85f9d978-txpwv" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "11.626150340744609" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG { 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "ds-idrepo-0" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "ds-cts-2" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "instance": "10.142.0.107:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "instance": "10.142.0.43:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0.3741666666666997" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "ds-cts-0" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "ds-cts-2" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "30.266666666666666" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "instance": "10.142.0.107:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "instance": "10.142.0.43:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "100488456.4" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "3318.5666666666666" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "196.336" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "instance": "10.142.0.107:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "instance": "10.142.0.43:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0.07066666666666871" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG { 20:51:35 DEBUG { 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "data": { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:35 DEBUG }, 20:51:35 DEBUG "result": [ 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG { 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG "metric": { 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "ds-idrepo-0" 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG }, 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG "value": [ 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:35 DEBUG [...] 20:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:35 DEBUG }, 20:51:35 DEBUG 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG "pod": "ds-idrepo-2" 20:51:35 DEBUG "3340.633333333333" 20:51:35 DEBUG ] 20:51:35 DEBUG }, 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG "value": [ 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG "0" 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "ds-idrepo-0" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "ds-idrepo-2" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "3566194688" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG { 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "instance": "10.142.0.107:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "instance": "10.142.0.43:9100" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "4554110911488" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "ds-cts-0" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "ds-cts-2" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "ds-idrepo-0" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "ds-idrepo-2" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "3340.3333333333335" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "idm-b85f9d978-txpwv" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "ds-cts-0" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "ds-cts-2" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0.5381666666666669" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "0" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "3336.9333333333334" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729095" 20:51:35 INFO [http_cmd]: http status code OK 20:51:35 DEBUG --- status code --- 20:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:35 DEBUG --- http response --- 20:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:35 DEBUG ----- output ----- 20:51:35 DEBUG { 20:51:35 DEBUG "status": "success", 20:51:35 DEBUG "data": { 20:51:35 DEBUG "resultType": "vector", 20:51:35 DEBUG "result": [ 20:51:35 DEBUG { 20:51:35 DEBUG "metric": { 20:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG [...] 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG 20:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:51:35 DEBUG }, 20:51:35 DEBUG "value": [ 20:51:35 DEBUG 1713729095, 20:51:35 DEBUG "600331" 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG ] 20:51:35 DEBUG } 20:51:35 DEBUG } 20:51:36 INFO 20:51:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:51:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:51:36 INFO [loop_until]: OK (rc = 0) 20:51:36 DEBUG --- stdout --- 20:51:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21594m 8801Mi am-767f6b67b5-qcqz2 21811m 8993Mi am-767f6b67b5-vbltz 23803m 8880Mi ds-cts-0 19823m 12042Mi ds-cts-1 20030m 12145Mi ds-cts-2 20490m 11979Mi ds-idrepo-0 9472m 4027Mi ds-idrepo-1 9519m 3998Mi ds-idrepo-2 9877m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11428m 3310Mi 20:51:36 DEBUG --- stderr --- 20:51:36 DEBUG 20:51:36 INFO 20:51:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:51:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:51:36 INFO [loop_until]: OK (rc = 0) 20:51:36 DEBUG --- stdout --- 20:51:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6304Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10009m 33% 7868Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22217m 74% 11639Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3354Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20013m 67% 15347Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5308Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22263m 74% 11898Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19732m 66% 3224Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24780m 83% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 10411m 34% 7443Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19835m 66% 17242Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10142m 33% 8035Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20400m 68% 14694Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11867m 74% 13558Mi 23% 20:51:36 DEBUG --- stderr --- 20:51:36 DEBUG 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "11.612806190933103" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "instance": "10.142.0.107:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "instance": "10.142.0.43:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "100084009.53333333" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 ERROR [http_cmd]: ERROR 20:52:35 ERROR --- status code --- 20:52:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:52:35 ERROR --- http response --- 20:52:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:52:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-idrepo-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "142021792.48242113" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "instance": "10.142.0.107:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "instance": "10.142.0.43:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0.422366666666655" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-cts-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-cts-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-cts-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "31.8" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "instance": "10.142.0.107:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "instance": "10.142.0.43:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "100085228.03333333" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "3331.8333333333335" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "196.336" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-idrepo-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "instance": "10.142.0.107:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "instance": "10.142.0.43:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0.0876666666666665" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-idrepo-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "3331.233333333333" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "3566628864" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (80 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-idrepo-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "2843.30521849809" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG { 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-cts-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-cts-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "instance": "10.142.0.107:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "instance": "10.142.0.43:9100" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "4559210429440" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-idrepo-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG ] 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG } 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG } 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "3331.1" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-idrepo-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "5.258459588189713" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729155" 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (80 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-idrepo-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-idrepo-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:35 INFO [http_cmd]: http status code OK 20:52:35 DEBUG --- status code --- 20:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:35 DEBUG --- http response --- 20:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:35 DEBUG ----- output ----- 20:52:35 DEBUG { 20:52:35 DEBUG "status": "success", 20:52:35 DEBUG "data": { 20:52:35 DEBUG "resultType": "vector", 20:52:35 DEBUG "result": [ 20:52:35 DEBUG { 20:52:35 DEBUG "metric": { 20:52:35 DEBUG "pod": "ds-cts-0" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG [...] 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG 20:52:35 DEBUG "pod": "ds-cts-2" 20:52:35 DEBUG }, 20:52:35 DEBUG "value": [ 20:52:35 DEBUG 1713729155, 20:52:35 DEBUG "0.5734333333333335" 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG ] 20:52:35 DEBUG } 20:52:35 DEBUG } 20:52:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729156" 20:52:36 INFO [http_cmd]: http status code OK 20:52:36 DEBUG --- status code --- 20:52:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:36 DEBUG --- http response --- 20:52:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:36 DEBUG ----- output ----- 20:52:36 DEBUG { 20:52:36 DEBUG "status": "success", 20:52:36 DEBUG "data": { 20:52:36 DEBUG "resultType": "vector", 20:52:36 DEBUG "result": [ 20:52:36 DEBUG { 20:52:36 DEBUG "metric": { 20:52:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:36 DEBUG }, 20:52:36 DEBUG "value": [ 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG [...] 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:36 DEBUG }, 20:52:36 DEBUG "value": [ 20:52:36 DEBUG 1713729156, 20:52:36 DEBUG "3332.7" 20:52:36 DEBUG ] 20:52:36 DEBUG } 20:52:36 DEBUG ] 20:52:36 DEBUG } 20:52:36 DEBUG } 20:52:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729156" 20:52:36 INFO [http_cmd]: http status code OK 20:52:36 DEBUG --- status code --- 20:52:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:36 DEBUG --- http response --- 20:52:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:36 DEBUG ----- output ----- 20:52:36 DEBUG { 20:52:36 DEBUG "status": "success", 20:52:36 DEBUG "data": { 20:52:36 DEBUG "resultType": "vector", 20:52:36 DEBUG "result": [ 20:52:36 DEBUG { 20:52:36 DEBUG "metric": { 20:52:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:52:36 DEBUG }, 20:52:36 DEBUG "value": [ 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG [...] 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG 20:52:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:52:36 DEBUG }, 20:52:36 DEBUG "value": [ 20:52:36 DEBUG 1713729156, 20:52:36 DEBUG "599864" 20:52:36 DEBUG ] 20:52:36 DEBUG } 20:52:36 DEBUG ] 20:52:36 DEBUG } 20:52:36 DEBUG } 20:52:36 INFO 20:52:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:52:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:52:36 INFO [loop_until]: OK (rc = 0) 20:52:36 DEBUG --- stdout --- 20:52:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22499m 8801Mi am-767f6b67b5-qcqz2 21875m 8993Mi am-767f6b67b5-vbltz 24337m 8870Mi ds-cts-0 20281m 12039Mi ds-cts-1 20135m 12151Mi ds-cts-2 20323m 11983Mi ds-idrepo-0 9525m 4026Mi ds-idrepo-1 9521m 3999Mi ds-idrepo-2 9913m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11736m 3310Mi 20:52:36 DEBUG --- stderr --- 20:52:36 DEBUG 20:52:36 INFO 20:52:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:52:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:52:36 INFO [loop_until]: OK (rc = 0) 20:52:36 DEBUG --- stdout --- 20:52:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10069m 33% 7862Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22065m 73% 11631Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3368Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20075m 67% 15348Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5298Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22333m 74% 11894Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19677m 65% 3218Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24634m 82% 11171Mi 10% gke-xlou-cdm-ds-0f601751-j488 10366m 34% 7454Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19850m 66% 17266Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10087m 33% 8044Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19646m 65% 14698Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11866m 74% 13552Mi 23% 20:52:36 DEBUG --- stderr --- 20:52:36 DEBUG 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "instance": "10.142.0.107:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "instance": "10.142.0.43:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "100401399.3" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "11.638183998778805" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729215" 20:53:35 ERROR [http_cmd]: ERROR 20:53:35 ERROR --- status code --- 20:53:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:53:35 ERROR --- http response --- 20:53:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:53:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "ds-idrepo-0" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "instance": "10.142.0.107:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "instance": "10.142.0.43:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0.41763333333334834" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "ds-cts-0" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "ds-cts-2" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "30.599999999999998" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "ds-idrepo-0" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "ds-cts-2" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "3358.1" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "instance": "10.142.0.107:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "instance": "10.142.0.43:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "100401874.36666667" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "196.336" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "ds-idrepo-0" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "ds-idrepo-2" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "3570638848" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "instance": "10.142.0.107:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "instance": "10.142.0.43:9100" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0.08266666666667295" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "ds-idrepo-0" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "ds-idrepo-2" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "3326.9333333333334" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "ds-idrepo-0" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "ds-idrepo-2" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "pod": "ds-idrepo-0" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG [...] 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "0" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729215" 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG { 20:53:35 DEBUG { 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG "data": { 20:53:35 INFO [http_cmd]: http status code OK 20:53:35 DEBUG "data": { 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG --- status code --- 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG "result": [ 20:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:35 DEBUG "result": [ 20:53:35 DEBUG { 20:53:35 DEBUG --- http response --- 20:53:35 DEBUG { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG "metric": { 20:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:35 DEBUG "pod": "ds-cts-0" 20:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:35 DEBUG ----- output ----- 20:53:35 DEBUG }, 20:53:35 DEBUG }, 20:53:35 DEBUG { 20:53:35 DEBUG "value": [ 20:53:35 DEBUG "value": [ 20:53:35 DEBUG "status": "success", 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "data": { 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "resultType": "vector", 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "result": [ 20:53:35 DEBUG [...] 20:53:35 DEBUG [...] 20:53:35 DEBUG { 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "metric": { 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG "instance": "10.142.0.107:9100" 20:53:35 DEBUG 20:53:35 DEBUG 20:53:35 DEBUG }, 20:53:35 DEBUG "pod": "ds-cts-2" 20:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:35 DEBUG "value": [ 20:53:35 DEBUG }, 20:53:35 DEBUG }, 20:53:35 DEBUG 20:53:35 DEBUG "value": [ 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG 20:53:35 DEBUG "0" 20:53:35 DEBUG "3323.633333333333" 20:53:35 DEBUG [...] 20:53:35 DEBUG ] 20:53:35 DEBUG ] 20:53:35 DEBUG 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 DEBUG 20:53:35 DEBUG ] 20:53:35 DEBUG ] 20:53:35 DEBUG 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 DEBUG "instance": "10.142.0.43:9100" 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 DEBUG }, 20:53:35 DEBUG "value": [ 20:53:35 DEBUG 1713729215, 20:53:35 DEBUG "4564642071552" 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG ] 20:53:35 DEBUG } 20:53:35 DEBUG } 20:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729215" 20:53:36 INFO [http_cmd]: http status code OK 20:53:36 DEBUG --- status code --- 20:53:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:36 DEBUG --- http response --- 20:53:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:53:36 DEBUG ----- output ----- 20:53:36 DEBUG { 20:53:36 DEBUG "status": "success", 20:53:36 DEBUG "data": { 20:53:36 DEBUG "resultType": "vector", 20:53:36 DEBUG "result": [ 20:53:36 DEBUG { 20:53:36 DEBUG "metric": { 20:53:36 DEBUG "pod": "ds-idrepo-0" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG [...] 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 1713729215, 20:53:36 DEBUG "0" 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG } 20:53:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729216" 20:53:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729216" 20:53:36 INFO [http_cmd]: http status code OK 20:53:36 DEBUG --- status code --- 20:53:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:36 DEBUG --- http response --- 20:53:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:53:36 DEBUG ----- output ----- 20:53:36 DEBUG { 20:53:36 DEBUG "status": "success", 20:53:36 DEBUG "data": { 20:53:36 DEBUG "resultType": "vector", 20:53:36 DEBUG "result": [ 20:53:36 DEBUG { 20:53:36 DEBUG "metric": { 20:53:36 DEBUG "pod": "ds-idrepo-0" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG [...] 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 1713729216, 20:53:36 DEBUG "0" 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG } 20:53:36 INFO [http_cmd]: http status code OK 20:53:36 DEBUG --- status code --- 20:53:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:36 DEBUG --- http response --- 20:53:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:36 DEBUG ----- output ----- 20:53:36 DEBUG { 20:53:36 DEBUG "status": "success", 20:53:36 DEBUG "data": { 20:53:36 DEBUG "resultType": "vector", 20:53:36 DEBUG "result": [ 20:53:36 DEBUG { 20:53:36 DEBUG "metric": { 20:53:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG [...] 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 1713729216, 20:53:36 DEBUG "3326.9666666666667" 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG } 20:53:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729216" 20:53:36 INFO [http_cmd]: http status code OK 20:53:36 DEBUG --- status code --- 20:53:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:36 DEBUG --- http response --- 20:53:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:36 DEBUG ----- output ----- 20:53:36 DEBUG { 20:53:36 DEBUG "status": "success", 20:53:36 DEBUG "data": { 20:53:36 DEBUG "resultType": "vector", 20:53:36 DEBUG "result": [ 20:53:36 DEBUG { 20:53:36 DEBUG "metric": { 20:53:36 DEBUG "pod": "ds-cts-0" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG [...] 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG "pod": "ds-cts-2" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 1713729216, 20:53:36 DEBUG "0.5358333333333338" 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG } 20:53:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729216" 20:53:36 INFO [http_cmd]: http status code OK 20:53:36 DEBUG --- status code --- 20:53:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:36 DEBUG --- http response --- 20:53:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:36 DEBUG ----- output ----- 20:53:36 DEBUG { 20:53:36 DEBUG "status": "success", 20:53:36 DEBUG "data": { 20:53:36 DEBUG "resultType": "vector", 20:53:36 DEBUG "result": [ 20:53:36 DEBUG { 20:53:36 DEBUG "metric": { 20:53:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG [...] 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG 20:53:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:53:36 DEBUG }, 20:53:36 DEBUG "value": [ 20:53:36 DEBUG 1713729216, 20:53:36 DEBUG "600218.5" 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG ] 20:53:36 DEBUG } 20:53:36 DEBUG } 20:53:36 INFO 20:53:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:53:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:53:36 INFO [loop_until]: OK (rc = 0) 20:53:36 DEBUG --- stdout --- 20:53:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22068m 8799Mi am-767f6b67b5-qcqz2 22038m 8993Mi am-767f6b67b5-vbltz 24054m 8871Mi ds-cts-0 20010m 12044Mi ds-cts-1 19934m 12153Mi ds-cts-2 20773m 11987Mi ds-idrepo-0 9679m 4027Mi ds-idrepo-1 9184m 3992Mi ds-idrepo-2 9599m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1181Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11469m 3313Mi 20:53:36 DEBUG --- stderr --- 20:53:36 DEBUG 20:53:36 INFO 20:53:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:53:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:53:36 INFO [loop_until]: OK (rc = 0) 20:53:36 DEBUG --- stdout --- 20:53:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6299Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10202m 34% 7865Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22018m 73% 11636Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3363Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20148m 67% 15357Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5322Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22287m 74% 11892Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18985m 63% 3214Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24742m 82% 11186Mi 10% gke-xlou-cdm-ds-0f601751-j488 10371m 34% 7454Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19878m 66% 17272Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10079m 33% 8034Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20262m 67% 14693Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11862m 74% 13554Mi 23% 20:53:36 DEBUG --- stderr --- 20:53:36 DEBUG 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "instance": "10.142.0.107:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "instance": "10.142.0.43:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "100437629.83333333" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729275" 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "11.677977466379286" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "ds-idrepo-1" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729275" 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "ds-idrepo-2" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "9079.270455531452" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 ERROR [http_cmd]: ERROR 20:54:35 ERROR --- status code --- 20:54:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:54:35 ERROR --- http response --- 20:54:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:54:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "instance": "10.142.0.107:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "instance": "10.142.0.43:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "0.403866666666651" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "ds-cts-0" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "ds-cts-2" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "30.9" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "ds-idrepo-0" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "ds-cts-2" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "0" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "3330.5666666666666" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "ds-idrepo-0" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "ds-idrepo-2" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "0" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729275" 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "3571572736" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "instance": "10.142.0.107:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "instance": "10.142.0.43:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "100438676.69999999" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "196.336" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729275" 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "0" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "instance": "10.142.0.107:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "instance": "10.142.0.43:9100" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "0.07866666666667849" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729275" 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "ds-idrepo-0" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "ds-idrepo-2" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "0" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:35 INFO [http_cmd]: http status code OK 20:54:35 DEBUG --- status code --- 20:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:35 DEBUG --- http response --- 20:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:35 DEBUG ----- output ----- 20:54:35 DEBUG { 20:54:35 DEBUG "status": "success", 20:54:35 DEBUG "data": { 20:54:35 DEBUG "resultType": "vector", 20:54:35 DEBUG "result": [ 20:54:35 DEBUG { 20:54:35 DEBUG "metric": { 20:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG [...] 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG 20:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:35 DEBUG }, 20:54:35 DEBUG "value": [ 20:54:35 DEBUG 1713729275, 20:54:35 DEBUG "3326.1666666666665" 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG ] 20:54:35 DEBUG } 20:54:35 DEBUG } 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "ds-idrepo-0" 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:36 DEBUG }, 20:54:36 DEBUG "pod": "ds-idrepo-1" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "pod": "ds-idrepo-2" 20:54:36 DEBUG 20:54:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:36 DEBUG }, 20:54:36 DEBUG "pod": "ds-idrepo-2" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "0" 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "3329.2999999999997" 20:54:36 DEBUG ] 20:54:36 DEBUG "2813.3937237888645" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "ds-cts-0" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "pod": "ds-cts-2" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "0" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729276" 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "instance": "10.142.0.107:9100" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "instance": "10.142.0.43:9100" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "4569796393984" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "3329.266666666667" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "ds-idrepo-1" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "pod": "ds-idrepo-2" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "5.589359214154666" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "ds-idrepo-1" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "pod": "ds-idrepo-2" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "0" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "ds-cts-0" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "pod": "ds-cts-2" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "0.5838666666666654" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729276" 20:54:36 INFO [http_cmd]: http status code OK 20:54:36 DEBUG --- status code --- 20:54:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:36 DEBUG --- http response --- 20:54:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:36 DEBUG ----- output ----- 20:54:36 DEBUG { 20:54:36 DEBUG "status": "success", 20:54:36 DEBUG "data": { 20:54:36 DEBUG "resultType": "vector", 20:54:36 DEBUG "result": [ 20:54:36 DEBUG { 20:54:36 DEBUG "metric": { 20:54:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG [...] 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG 20:54:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:54:36 DEBUG }, 20:54:36 DEBUG "value": [ 20:54:36 DEBUG 1713729276, 20:54:36 DEBUG "599937" 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG ] 20:54:36 DEBUG } 20:54:36 DEBUG } 20:54:36 INFO 20:54:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:54:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:54:36 INFO [loop_until]: OK (rc = 0) 20:54:36 DEBUG --- stdout --- 20:54:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22206m 8800Mi am-767f6b67b5-qcqz2 22680m 8994Mi am-767f6b67b5-vbltz 24122m 8866Mi ds-cts-0 20342m 12053Mi ds-cts-1 19928m 12153Mi ds-cts-2 19695m 12021Mi ds-idrepo-0 9891m 4026Mi ds-idrepo-1 9518m 3992Mi ds-idrepo-2 9595m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11469m 3316Mi 20:54:36 DEBUG --- stderr --- 20:54:36 DEBUG 20:54:36 INFO 20:54:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:54:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:54:36 INFO [loop_until]: OK (rc = 0) 20:54:36 DEBUG --- stdout --- 20:54:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9671m 32% 7867Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22041m 73% 11628Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 77m 0% 3367Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20120m 67% 15342Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5311Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22271m 74% 11900Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19711m 66% 3214Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24636m 82% 11187Mi 10% gke-xlou-cdm-ds-0f601751-j488 10359m 34% 7437Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19849m 66% 17276Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10083m 33% 8034Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20155m 67% 14727Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11863m 74% 13555Mi 23% 20:54:36 DEBUG --- stderr --- 20:54:36 DEBUG 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729335" 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729335" 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "pod": "idm-b85f9d978-txpwv" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "6904.68785484343" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729335" 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "instance": "10.142.0.107:9100" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "instance": "10.142.0.43:9100" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "100692404.83333334" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729335" 20:55:35 ERROR [http_cmd]: ERROR 20:55:35 ERROR --- status code --- 20:55:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:55:35 ERROR --- http response --- 20:55:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:55:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "11.728132706159718" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729335" 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729335" 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "3347" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "instance": "10.142.0.107:9100" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "instance": "10.142.0.43:9100" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "0.40390000000000625" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729335" 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729335" 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729335" 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "pod": "ds-cts-0" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "pod": "ds-cts-2" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "31.366666666666667" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "pod": "ds-idrepo-0" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "pod": "ds-cts-2" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "0" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729335" 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729335" 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "pod": "ds-idrepo-0" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "pod": "ds-idrepo-2" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "0" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "3574464512" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729335" 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "196.336" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729335" 20:55:35 INFO [http_cmd]: http status code OK 20:55:35 DEBUG --- status code --- 20:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:35 DEBUG --- http response --- 20:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:35 DEBUG ----- output ----- 20:55:35 DEBUG { 20:55:35 DEBUG "status": "success", 20:55:35 DEBUG "data": { 20:55:35 DEBUG "resultType": "vector", 20:55:35 DEBUG "result": [ 20:55:35 DEBUG { 20:55:35 DEBUG "metric": { 20:55:35 DEBUG "instance": "10.142.0.107:9100" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG [...] 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG 20:55:35 DEBUG "instance": "10.142.0.43:9100" 20:55:35 DEBUG }, 20:55:35 DEBUG "value": [ 20:55:35 DEBUG 1713729335, 20:55:35 DEBUG "100693180.76666667" 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG ] 20:55:35 DEBUG } 20:55:35 DEBUG } 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "3343.0333333333333" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729336" 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "ds-idrepo-0" 20:55:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:36 DEBUG }, 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "pod": "ds-idrepo-2" 20:55:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:36 DEBUG }, 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "0" 20:55:36 DEBUG "0" 20:55:36 DEBUG ] 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "instance": "10.142.0.107:9100" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "instance": "10.142.0.43:9100" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "0.07833333333333596" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729336" 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "ds-idrepo-0" 20:55:36 DEBUG "instance": "10.142.0.107:9100" 20:55:36 DEBUG }, 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG [...] 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG { 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "pod": "ds-idrepo-2" 20:55:36 DEBUG "instance": "10.142.0.43:9100" 20:55:36 DEBUG "data": { 20:55:36 DEBUG }, 20:55:36 DEBUG }, 20:55:36 DEBUG "resultType": "vector", 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG "value": [ 20:55:36 DEBUG "value": [ 20:55:36 DEBUG "result": [ 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG { 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG "0" 20:55:36 DEBUG "4574987881472" 20:55:36 DEBUG "metric": { 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG ] 20:55:36 DEBUG ] 20:55:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:55:36 DEBUG }, 20:55:36 DEBUG ] 20:55:36 DEBUG ] 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG "value": [ 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 DEBUG { 20:55:36 DEBUG 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG "data": { 20:55:36 DEBUG 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG [...] 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG "result": [ 20:55:36 DEBUG 20:55:36 DEBUG { 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:36 DEBUG 20:55:36 DEBUG "metric": { 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG 20:55:36 DEBUG "pod": "ds-idrepo-1" 20:55:36 DEBUG { 20:55:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:36 DEBUG }, 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG "data": { 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG 20:55:36 DEBUG "result": [ 20:55:36 DEBUG "3341.7" 20:55:36 DEBUG 20:55:36 DEBUG { 20:55:36 DEBUG ] 20:55:36 DEBUG [...] 20:55:36 DEBUG "metric": { 20:55:36 DEBUG } 20:55:36 DEBUG 20:55:36 DEBUG "pod": "ds-cts-0" 20:55:36 DEBUG ] 20:55:36 DEBUG 20:55:36 DEBUG }, 20:55:36 DEBUG } 20:55:36 DEBUG 20:55:36 DEBUG "value": [ 20:55:36 DEBUG } 20:55:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:55:36 DEBUG 20:55:36 DEBUG }, 20:55:36 DEBUG 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG [...] 20:55:36 DEBUG "3295.884742125834" 20:55:36 DEBUG 20:55:36 DEBUG ] 20:55:36 DEBUG 20:55:36 DEBUG } 20:55:36 DEBUG 20:55:36 DEBUG ] 20:55:36 DEBUG "pod": "ds-cts-2" 20:55:36 DEBUG } 20:55:36 DEBUG }, 20:55:36 DEBUG } 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "0" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "ds-idrepo-1" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "0.0028145834929867116" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "3324.1666666666665" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "ds-idrepo-1" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "0" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "ds-cts-0" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "pod": "ds-cts-2" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "0.5423333333333327" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729336" 20:55:36 INFO [http_cmd]: http status code OK 20:55:36 DEBUG --- status code --- 20:55:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:36 DEBUG --- http response --- 20:55:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:36 DEBUG ----- output ----- 20:55:36 DEBUG { 20:55:36 DEBUG "status": "success", 20:55:36 DEBUG "data": { 20:55:36 DEBUG "resultType": "vector", 20:55:36 DEBUG "result": [ 20:55:36 DEBUG { 20:55:36 DEBUG "metric": { 20:55:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG [...] 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG 20:55:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:55:36 DEBUG }, 20:55:36 DEBUG "value": [ 20:55:36 DEBUG 1713729336, 20:55:36 DEBUG "599670.5" 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG ] 20:55:36 DEBUG } 20:55:36 DEBUG } 20:55:36 INFO 20:55:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:55:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:55:36 INFO [loop_until]: OK (rc = 0) 20:55:36 DEBUG --- stdout --- 20:55:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22135m 8799Mi am-767f6b67b5-qcqz2 22703m 8996Mi am-767f6b67b5-vbltz 24618m 8865Mi ds-cts-0 20369m 12123Mi ds-cts-1 20149m 12159Mi ds-cts-2 20659m 12019Mi ds-idrepo-0 9607m 4027Mi ds-idrepo-1 9542m 3993Mi ds-idrepo-2 9843m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11526m 3318Mi 20:55:36 DEBUG --- stderr --- 20:55:36 DEBUG 20:55:36 INFO 20:55:36 INFO [loop_until]: kubectl --namespace=xlou top node 20:55:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:55:37 INFO [loop_until]: OK (rc = 0) 20:55:37 DEBUG --- stdout --- 20:55:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6305Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10051m 33% 7878Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22039m 73% 11620Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20135m 67% 15410Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5309Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22336m 74% 11913Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19701m 65% 3217Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24822m 83% 11174Mi 10% gke-xlou-cdm-ds-0f601751-j488 10373m 34% 7447Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20030m 67% 17280Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10154m 34% 8040Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20368m 68% 14728Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11875m 74% 13553Mi 23% 20:55:37 DEBUG --- stderr --- 20:55:37 DEBUG 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729395" 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "instance": "10.142.0.107:9100" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "instance": "10.142.0.43:9100" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "100451570.23333333" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729395" 20:56:35 ERROR [http_cmd]: ERROR 20:56:35 ERROR --- status code --- 20:56:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:56:35 ERROR --- http response --- 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729395" 20:56:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:56:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729395" 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "pod": "ds-idrepo-1" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "320772213.84845597" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "11.594502405246299" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729395" 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729395" 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "3332.5666666666666" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "pod": "ds-cts-0" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "pod": "ds-cts-2" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "29.966666666666665" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729395" 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729395" 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729395" 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "pod": "ds-idrepo-0" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "pod": "ds-cts-2" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "0" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "instance": "10.142.0.107:9100" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "instance": "10.142.0.43:9100" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "0.4048666666666937" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "3579703296" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729395" 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729395" 20:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729395" 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 INFO [http_cmd]: http status code OK 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG --- status code --- 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG --- http response --- 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "pod": "ds-idrepo-0" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "pod": "ds-idrepo-2" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "0" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:35 DEBUG ----- output ----- 20:56:35 DEBUG { 20:56:35 DEBUG "status": "success", 20:56:35 DEBUG "data": { 20:56:35 DEBUG "resultType": "vector", 20:56:35 DEBUG "result": [ 20:56:35 DEBUG { 20:56:35 DEBUG "metric": { 20:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG [...] 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG 20:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:35 DEBUG }, 20:56:35 DEBUG "value": [ 20:56:35 DEBUG 1713729395, 20:56:35 DEBUG "196.336" 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG ] 20:56:35 DEBUG } 20:56:35 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729396" 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "instance": "10.142.0.107:9100" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "instance": "10.142.0.43:9100" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "100451582.23333333" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "ds-idrepo-0" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "ds-idrepo-2" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "0" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "3336.1666666666665" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "instance": "10.142.0.107:9100" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "instance": "10.142.0.43:9100" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "0.0793333333333313" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "0" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "ds-cts-0" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "ds-cts-2" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "0" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "instance": "10.142.0.107:9100" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "instance": "10.142.0.43:9100" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "4579996017664" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "ds-idrepo-0" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "ds-idrepo-2" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "0" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "3336.4333333333334" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "ds-idrepo-1" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "557.2031016188273" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "ds-idrepo-1" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "22.306390014963924" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "ds-cts-0" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "ds-cts-2" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "0.48420000000000035" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "3335.8333333333335" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "ds-idrepo-1" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "0" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729396" 20:56:36 INFO [http_cmd]: http status code OK 20:56:36 DEBUG --- status code --- 20:56:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:36 DEBUG --- http response --- 20:56:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:36 DEBUG ----- output ----- 20:56:36 DEBUG { 20:56:36 DEBUG "status": "success", 20:56:36 DEBUG "data": { 20:56:36 DEBUG "resultType": "vector", 20:56:36 DEBUG "result": [ 20:56:36 DEBUG { 20:56:36 DEBUG "metric": { 20:56:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG [...] 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG 20:56:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:56:36 DEBUG }, 20:56:36 DEBUG "value": [ 20:56:36 DEBUG 1713729396, 20:56:36 DEBUG "599656" 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG ] 20:56:36 DEBUG } 20:56:36 DEBUG } 20:56:36 INFO 20:56:36 INFO [loop_until]: kubectl --namespace=xlou top pods 20:56:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:56:36 INFO [loop_until]: OK (rc = 0) 20:56:36 DEBUG --- stdout --- 20:56:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22520m 8803Mi am-767f6b67b5-qcqz2 21915m 8994Mi am-767f6b67b5-vbltz 24415m 8882Mi ds-cts-0 20248m 12123Mi ds-cts-1 19461m 12163Mi ds-cts-2 20561m 12019Mi ds-idrepo-0 9619m 4027Mi ds-idrepo-1 9607m 3993Mi ds-idrepo-2 9844m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11427m 3320Mi 20:56:36 DEBUG --- stderr --- 20:56:36 DEBUG 20:56:37 INFO 20:56:37 INFO [loop_until]: kubectl --namespace=xlou top node 20:56:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:56:37 INFO [loop_until]: OK (rc = 0) 20:56:37 DEBUG --- stdout --- 20:56:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10028m 33% 7870Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22089m 73% 11634Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3359Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20163m 67% 15431Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 5311Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22290m 74% 11896Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19728m 66% 3222Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24847m 83% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 10345m 34% 7436Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19736m 66% 17274Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10107m 33% 8036Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20445m 68% 14739Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11862m 74% 13559Mi 23% 20:56:37 DEBUG --- stderr --- 20:56:37 DEBUG 20:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729455" 20:57:35 INFO [http_cmd]: http status code OK 20:57:35 DEBUG --- status code --- 20:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:35 DEBUG --- http response --- 20:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:35 DEBUG ----- output ----- 20:57:35 DEBUG { 20:57:35 DEBUG "status": "success", 20:57:35 DEBUG "data": { 20:57:35 DEBUG "resultType": "vector", 20:57:35 DEBUG "result": [ 20:57:35 DEBUG { 20:57:35 DEBUG "metric": { 20:57:35 DEBUG "instance": "10.142.0.107:9100" 20:57:35 DEBUG }, 20:57:35 DEBUG "value": [ 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG [...] 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG "instance": "10.142.0.43:9100" 20:57:35 DEBUG }, 20:57:35 DEBUG "value": [ 20:57:35 DEBUG 1713729455, 20:57:35 DEBUG "100047400.16666666" 20:57:35 DEBUG ] 20:57:35 DEBUG } 20:57:35 DEBUG ] 20:57:35 DEBUG } 20:57:35 DEBUG } 20:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729455" 20:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729455" 20:57:35 ERROR [http_cmd]: ERROR 20:57:35 ERROR --- status code --- 20:57:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:57:35 ERROR --- http response --- 20:57:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:57:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:57:35 INFO [http_cmd]: http status code OK 20:57:35 DEBUG --- status code --- 20:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:35 DEBUG --- http response --- 20:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:35 DEBUG ----- output ----- 20:57:35 DEBUG { 20:57:35 DEBUG "status": "success", 20:57:35 DEBUG "data": { 20:57:35 DEBUG "resultType": "vector", 20:57:35 DEBUG "result": [ 20:57:35 DEBUG { 20:57:35 DEBUG "metric": { 20:57:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:57:35 DEBUG }, 20:57:35 DEBUG "value": [ 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG [...] 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:57:35 DEBUG }, 20:57:35 DEBUG "value": [ 20:57:35 DEBUG 1713729455, 20:57:35 DEBUG "11.643614157636387" 20:57:35 DEBUG ] 20:57:35 DEBUG } 20:57:35 DEBUG ] 20:57:35 DEBUG } 20:57:35 DEBUG } 20:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729455" 20:57:35 INFO [http_cmd]: http status code OK 20:57:35 DEBUG --- status code --- 20:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:35 DEBUG --- http response --- 20:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:57:35 DEBUG ----- output ----- 20:57:35 DEBUG { 20:57:35 DEBUG "status": "success", 20:57:35 DEBUG "data": { 20:57:35 DEBUG "resultType": "vector", 20:57:35 DEBUG "result": [ 20:57:35 DEBUG { 20:57:35 DEBUG "metric": { 20:57:35 DEBUG "pod": "ds-idrepo-0" 20:57:35 DEBUG }, 20:57:35 DEBUG "value": [ 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG [...] 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG 20:57:35 DEBUG "pod": "ds-idrepo-2" 20:57:35 DEBUG }, 20:57:35 DEBUG "value": [ 20:57:35 DEBUG 1713729455, 20:57:35 DEBUG "0" 20:57:35 DEBUG ] 20:57:35 DEBUG } 20:57:35 DEBUG ] 20:57:35 DEBUG } 20:57:35 DEBUG } 20:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729455" 20:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729455" 20:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729455" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729455, 20:57:36 DEBUG "3314.0666666666666" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "ds-cts-0" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-cts-2" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729455, 20:57:36 DEBUG "31.366666666666667" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:36 DEBUG "result": [ 20:57:36 DEBUG "result": [ 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG { 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "pod": "ds-idrepo-0" 20:57:36 DEBUG "instance": "10.142.0.107:9100" 20:57:36 DEBUG "data": { 20:57:36 DEBUG }, 20:57:36 DEBUG }, 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "value": [ 20:57:36 DEBUG "value": [ 20:57:36 DEBUG "result": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG { 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "metric": { 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:57:36 DEBUG [...] 20:57:36 DEBUG [...] 20:57:36 DEBUG }, 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-cts-2" 20:57:36 DEBUG "instance": "10.142.0.43:9100" 20:57:36 DEBUG 20:57:36 DEBUG }, 20:57:36 DEBUG }, 20:57:36 DEBUG [...] 20:57:36 DEBUG "value": [ 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG 20:57:36 DEBUG "0" 20:57:36 DEBUG "0.41156666666668495" 20:57:36 DEBUG 20:57:36 DEBUG ] 20:57:36 DEBUG ] 20:57:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG }, 20:57:36 DEBUG ] 20:57:36 DEBUG ] 20:57:36 DEBUG "value": [ 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG "3583733760" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "instance": "10.142.0.107:9100" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "instance": "10.142.0.43:9100" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "100047901.43333334" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "ds-idrepo-0" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-idrepo-2" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "0" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "196.336" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "3333.1666666666665" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "ds-idrepo-0" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-idrepo-2" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "0" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "instance": "10.142.0.107:9100" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "instance": "10.142.0.43:9100" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "0.07700000000000197" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "0" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "ds-cts-0" 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "instance": "10.142.0.107:9100" 20:57:36 DEBUG }, 20:57:36 DEBUG "pod": "ds-idrepo-0" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-cts-2" 20:57:36 DEBUG 20:57:36 DEBUG "instance": "10.142.0.43:9100" 20:57:36 DEBUG }, 20:57:36 DEBUG "pod": "ds-idrepo-2" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "0" 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "4585261949952" 20:57:36 DEBUG ] 20:57:36 DEBUG "0" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:36 DEBUG "pod": "ds-idrepo-0" 20:57:36 DEBUG }, 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:36 DEBUG "pod": "ds-idrepo-2" 20:57:36 DEBUG }, 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "3331.633333333333" 20:57:36 DEBUG "0" 20:57:36 DEBUG ] 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "ds-idrepo-0" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-idrepo-1" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "2987.8814914143113" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "ds-cts-0" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-cts-2" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "0.5167666666666682" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "3334.7" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "599972.5" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729456" 20:57:36 INFO [http_cmd]: http status code OK 20:57:36 DEBUG --- status code --- 20:57:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:36 DEBUG --- http response --- 20:57:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:57:36 DEBUG ----- output ----- 20:57:36 DEBUG { 20:57:36 DEBUG "status": "success", 20:57:36 DEBUG "data": { 20:57:36 DEBUG "resultType": "vector", 20:57:36 DEBUG "result": [ 20:57:36 DEBUG { 20:57:36 DEBUG "metric": { 20:57:36 DEBUG "pod": "ds-idrepo-0" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG [...] 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG 20:57:36 DEBUG "pod": "ds-idrepo-1" 20:57:36 DEBUG }, 20:57:36 DEBUG "value": [ 20:57:36 DEBUG 1713729456, 20:57:36 DEBUG "0" 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG ] 20:57:36 DEBUG } 20:57:36 DEBUG } 20:57:37 INFO 20:57:37 INFO [loop_until]: kubectl --namespace=xlou top pods 20:57:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:57:37 INFO [loop_until]: OK (rc = 0) 20:57:37 DEBUG --- stdout --- 20:57:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22503m 8799Mi am-767f6b67b5-qcqz2 22178m 8994Mi am-767f6b67b5-vbltz 24298m 8871Mi ds-cts-0 20115m 12131Mi ds-cts-1 19972m 12169Mi ds-cts-2 19923m 12028Mi ds-idrepo-0 9615m 4027Mi ds-idrepo-1 9663m 3993Mi ds-idrepo-2 9854m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11742m 3324Mi 20:57:37 DEBUG --- stderr --- 20:57:37 DEBUG 20:57:37 INFO 20:57:37 INFO [loop_until]: kubectl --namespace=xlou top node 20:57:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:57:37 INFO [loop_until]: OK (rc = 0) 20:57:37 DEBUG --- stdout --- 20:57:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10013m 33% 7868Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22103m 74% 11641Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19455m 65% 15440Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5308Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22370m 74% 11900Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19761m 66% 3221Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24842m 83% 11183Mi 10% gke-xlou-cdm-ds-0f601751-j488 10067m 33% 7443Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19935m 66% 17281Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10149m 33% 8031Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20343m 68% 14744Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11876m 74% 13559Mi 23% 20:57:37 DEBUG --- stderr --- 20:57:37 DEBUG 20:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729515" 20:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729515" 20:58:35 INFO [http_cmd]: http status code OK 20:58:35 DEBUG --- status code --- 20:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:35 DEBUG --- http response --- 20:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:35 DEBUG ----- output ----- 20:58:35 DEBUG { 20:58:35 DEBUG "status": "success", 20:58:35 DEBUG "data": { 20:58:35 DEBUG "resultType": "vector", 20:58:35 DEBUG "result": [ 20:58:35 DEBUG { 20:58:35 DEBUG "metric": { 20:58:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:58:35 DEBUG }, 20:58:35 DEBUG "value": [ 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG [...] 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:58:35 DEBUG }, 20:58:35 DEBUG "value": [ 20:58:35 DEBUG 1713729515, 20:58:35 DEBUG "11.646521214515419" 20:58:35 DEBUG ] 20:58:35 DEBUG } 20:58:35 DEBUG ] 20:58:35 DEBUG } 20:58:35 DEBUG } 20:58:35 INFO [http_cmd]: http status code OK 20:58:35 DEBUG --- status code --- 20:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:35 DEBUG --- http response --- 20:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:35 DEBUG ----- output ----- 20:58:35 DEBUG { 20:58:35 DEBUG "status": "success", 20:58:35 DEBUG "data": { 20:58:35 DEBUG "resultType": "vector", 20:58:35 DEBUG "result": [ 20:58:35 DEBUG { 20:58:35 DEBUG "metric": { 20:58:35 DEBUG "instance": "10.142.0.107:9100" 20:58:35 DEBUG }, 20:58:35 DEBUG "value": [ 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG [...] 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG "instance": "10.142.0.43:9100" 20:58:35 DEBUG }, 20:58:35 DEBUG "value": [ 20:58:35 DEBUG 1713729515, 20:58:35 DEBUG "100230625.56666666" 20:58:35 DEBUG ] 20:58:35 DEBUG } 20:58:35 DEBUG ] 20:58:35 DEBUG } 20:58:35 DEBUG } 20:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729515" 20:58:35 ERROR [http_cmd]: ERROR 20:58:35 ERROR --- status code --- 20:58:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:58:35 ERROR --- http response --- 20:58:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:58:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729515" 20:58:35 INFO [http_cmd]: http status code OK 20:58:35 DEBUG --- status code --- 20:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:35 DEBUG --- http response --- 20:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:58:35 DEBUG ----- output ----- 20:58:35 DEBUG { 20:58:35 DEBUG "status": "success", 20:58:35 DEBUG "data": { 20:58:35 DEBUG "resultType": "vector", 20:58:35 DEBUG "result": [ 20:58:35 DEBUG { 20:58:35 DEBUG "metric": { 20:58:35 DEBUG "pod": "ds-idrepo-1" 20:58:35 DEBUG }, 20:58:35 DEBUG "value": [ 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG [...] 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG 20:58:35 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 20:58:35 DEBUG }, 20:58:35 DEBUG "value": [ 20:58:35 DEBUG 1713729515, 20:58:35 DEBUG "81498.13744490972" 20:58:35 DEBUG ] 20:58:35 DEBUG } 20:58:35 DEBUG ] 20:58:35 DEBUG } 20:58:35 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "3335.133333333333" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-idrepo-0" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "ds-cts-2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-cts-0" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "ds-cts-2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "30.833333333333332" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-idrepo-0" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "ds-idrepo-2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "3586727936" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "instance": "10.142.0.107:9100" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "instance": "10.142.0.43:9100" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0.4063666666665824" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "196.336" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "instance": "10.142.0.107:9100" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "instance": "10.142.0.43:9100" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "100232042.53333333" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "3322.733333333333" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-idrepo-0" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "ds-idrepo-2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "instance": "10.142.0.107:9100" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "instance": "10.142.0.43:9100" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0.08299999999998516" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-cts-0" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "ds-cts-2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG { 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:58:36 DEBUG "data": { 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "instance": "10.142.0.107:9100" 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG }, 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "value": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-idrepo-1" 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:36 DEBUG }, 20:58:36 DEBUG 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG [...] 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG "instance": "10.142.0.43:9100" 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG }, 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:58:36 DEBUG "4590296443904" 20:58:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:36 DEBUG }, 20:58:36 DEBUG ] 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG } 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG ] 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "14.386798720716868" 20:58:36 DEBUG } 20:58:36 DEBUG "3325.2" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-idrepo-0" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "ds-idrepo-2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "3325.8" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-idrepo-1" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-idrepo-1" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "845.5430816981711" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "ds-cts-0" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "ds-cts-2" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "0.508299999999997" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729516" 20:58:36 INFO [http_cmd]: http status code OK 20:58:36 DEBUG --- status code --- 20:58:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:36 DEBUG --- http response --- 20:58:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:36 DEBUG ----- output ----- 20:58:36 DEBUG { 20:58:36 DEBUG "status": "success", 20:58:36 DEBUG "data": { 20:58:36 DEBUG "resultType": "vector", 20:58:36 DEBUG "result": [ 20:58:36 DEBUG { 20:58:36 DEBUG "metric": { 20:58:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG [...] 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG 20:58:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:58:36 DEBUG }, 20:58:36 DEBUG "value": [ 20:58:36 DEBUG 1713729516, 20:58:36 DEBUG "600648.5" 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG ] 20:58:36 DEBUG } 20:58:36 DEBUG } 20:58:37 INFO 20:58:37 INFO [loop_until]: kubectl --namespace=xlou top pods 20:58:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:58:37 INFO [loop_until]: OK (rc = 0) 20:58:37 DEBUG --- stdout --- 20:58:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21715m 8801Mi am-767f6b67b5-qcqz2 22274m 8993Mi am-767f6b67b5-vbltz 24278m 8878Mi ds-cts-0 20414m 12132Mi ds-cts-1 20372m 12170Mi ds-cts-2 20062m 12030Mi ds-idrepo-0 9719m 4026Mi ds-idrepo-1 9541m 3993Mi ds-idrepo-2 9918m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11653m 3327Mi 20:58:37 DEBUG --- stderr --- 20:58:37 DEBUG 20:58:37 INFO 20:58:37 INFO [loop_until]: kubectl --namespace=xlou top node 20:58:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:58:37 INFO [loop_until]: OK (rc = 0) 20:58:37 DEBUG --- stdout --- 20:58:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6307Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9984m 33% 7875Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22136m 74% 11630Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20079m 67% 15428Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5305Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22290m 74% 11900Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19645m 65% 3208Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24695m 82% 11195Mi 10% gke-xlou-cdm-ds-0f601751-j488 10317m 34% 7441Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19813m 66% 17287Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10096m 33% 8044Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20346m 68% 14749Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11855m 74% 13557Mi 23% 20:58:37 DEBUG --- stderr --- 20:58:37 DEBUG 20:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729575" 20:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729575" 20:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729575" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG { 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "instance": "10.142.0.107:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "instance": "10.142.0.43:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729575, 20:59:36 DEBUG "100227865.5" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729575, 20:59:36 DEBUG "11.608781164290226" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 ERROR [http_cmd]: ERROR 20:59:36 ERROR --- status code --- 20:59:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:59:36 ERROR --- http response --- 20:59:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:59:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "ds-idrepo-0" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "ds-cts-2" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "3328.1" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "ds-cts-0" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "ds-cts-2" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "32.13333333333333" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "3586867200" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "ds-idrepo-0" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "ds-idrepo-2" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "instance": "10.142.0.107:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "instance": "10.142.0.43:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0.4115999999999777" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "196.336" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "instance": "10.142.0.107:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG "instance": "10.142.0.43:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG "100228822.96666667" 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG ] 20:59:36 DEBUG { 20:59:36 DEBUG } 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG ] 20:59:36 DEBUG "data": { 20:59:36 DEBUG } 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG } 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "3326.233333333333" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "ds-idrepo-0" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "ds-idrepo-2" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "instance": "10.142.0.107:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "instance": "10.142.0.43:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0.07933333333333319" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "instance": "10.142.0.107:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "instance": "10.142.0.43:9100" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "4595534750720" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "ds-idrepo-0" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "ds-idrepo-2" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "3323.0333333333333" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "ds-cts-0" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "ds-cts-2" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "3322.9666666666667" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "2850.8336233351256" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "ds-cts-0" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "ds-cts-2" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "0.5388333333333322" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729576" 20:59:36 INFO [http_cmd]: http status code OK 20:59:36 DEBUG --- status code --- 20:59:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:36 DEBUG --- http response --- 20:59:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:36 DEBUG ----- output ----- 20:59:36 DEBUG { 20:59:36 DEBUG "status": "success", 20:59:36 DEBUG "data": { 20:59:36 DEBUG "resultType": "vector", 20:59:36 DEBUG "result": [ 20:59:36 DEBUG { 20:59:36 DEBUG "metric": { 20:59:36 DEBUG "pod": "am-767f6b67b5-9zw22" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG [...] 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG 20:59:36 DEBUG "pod": "am-767f6b67b5-vbltz" 20:59:36 DEBUG }, 20:59:36 DEBUG "value": [ 20:59:36 DEBUG 1713729576, 20:59:36 DEBUG "599963" 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG ] 20:59:36 DEBUG } 20:59:36 DEBUG } 20:59:37 INFO 20:59:37 INFO [loop_until]: kubectl --namespace=xlou top pods 20:59:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:59:37 INFO [loop_until]: OK (rc = 0) 20:59:37 DEBUG --- stdout --- 20:59:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22093m 8804Mi am-767f6b67b5-qcqz2 22034m 8995Mi am-767f6b67b5-vbltz 23318m 8874Mi ds-cts-0 20558m 12136Mi ds-cts-1 20321m 12221Mi ds-cts-2 19855m 12033Mi ds-idrepo-0 9641m 4026Mi ds-idrepo-1 9800m 3993Mi ds-idrepo-2 9713m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11564m 3328Mi 20:59:37 DEBUG --- stderr --- 20:59:37 DEBUG 20:59:37 INFO 20:59:37 INFO [loop_until]: kubectl --namespace=xlou top node 20:59:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:59:37 INFO [loop_until]: OK (rc = 0) 20:59:37 DEBUG --- stdout --- 20:59:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6300Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10027m 33% 7872Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21303m 71% 11628Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3362Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20359m 68% 15437Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5310Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22330m 74% 11905Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19644m 65% 3207Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24585m 82% 11175Mi 10% gke-xlou-cdm-ds-0f601751-j488 10345m 34% 7448Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19826m 66% 17330Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10126m 33% 8043Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20322m 68% 14757Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11843m 74% 13558Mi 23% 20:59:37 DEBUG --- stderr --- 20:59:37 DEBUG 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "instance": "10.142.0.107:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "instance": "10.142.0.43:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "100731393.53333333" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729636" 21:00:36 ERROR [http_cmd]: ERROR 21:00:36 ERROR --- status code --- 21:00:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:00:36 ERROR --- http response --- 21:00:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:00:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "11.694111583857161" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-cts-2" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-idrepo-2" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "3509.5" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-cts-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-cts-2" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "30.133333333333333" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "3589300224" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "instance": "10.142.0.107:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "instance": "10.142.0.43:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0.4223333333333604" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "196.7225" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "3329.233333333333" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-idrepo-2" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "instance": "10.142.0.107:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "instance": "10.142.0.43:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "100732432.93333334" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "instance": "10.142.0.107:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "instance": "10.142.0.43:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0.09433333333334415" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "instance": "10.142.0.107:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "instance": "10.142.0.43:9100" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "4600752600064" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-idrepo-2" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "3323.8" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-cts-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-cts-2" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0.4943333333333309" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-cts-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-cts-2" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "0" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "3329.2999999999997" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-cts-1" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "41329302.81138674" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "ds-idrepo-0" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "ds-cts-1" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "47622081.55804374" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729636" 21:00:36 INFO [http_cmd]: http status code OK 21:00:36 DEBUG --- status code --- 21:00:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:00:36 DEBUG --- http response --- 21:00:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:00:36 DEBUG ----- output ----- 21:00:36 DEBUG { 21:00:36 DEBUG "status": "success", 21:00:36 DEBUG "data": { 21:00:36 DEBUG "resultType": "vector", 21:00:36 DEBUG "result": [ 21:00:36 DEBUG { 21:00:36 DEBUG "metric": { 21:00:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG [...] 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG 21:00:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:00:36 DEBUG }, 21:00:36 DEBUG "value": [ 21:00:36 DEBUG 1713729636, 21:00:36 DEBUG "599578.5" 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG ] 21:00:36 DEBUG } 21:00:36 DEBUG } 21:00:37 INFO 21:00:37 INFO [loop_until]: kubectl --namespace=xlou top pods 21:00:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:00:37 INFO [loop_until]: OK (rc = 0) 21:00:37 DEBUG --- stdout --- 21:00:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22438m 8804Mi am-767f6b67b5-qcqz2 22353m 8995Mi am-767f6b67b5-vbltz 24651m 8873Mi ds-cts-0 20535m 12139Mi ds-cts-1 19642m 12219Mi ds-cts-2 20341m 12041Mi ds-idrepo-0 9922m 4027Mi ds-idrepo-1 9443m 3993Mi ds-idrepo-2 9833m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1181Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11728m 3331Mi 21:00:37 DEBUG --- stderr --- 21:00:37 DEBUG 21:00:37 INFO 21:00:37 INFO [loop_until]: kubectl --namespace=xlou top node 21:00:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:00:37 INFO [loop_until]: OK (rc = 0) 21:00:37 DEBUG --- stdout --- 21:00:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6294Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10034m 33% 7875Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22137m 74% 11621Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3365Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20095m 67% 15440Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5283Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22397m 75% 11901Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19771m 66% 3226Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24806m 83% 11180Mi 10% gke-xlou-cdm-ds-0f601751-j488 10347m 34% 7433Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19608m 65% 17339Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10083m 33% 8037Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20449m 68% 14749Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11863m 74% 13557Mi 23% 21:00:37 DEBUG --- stderr --- 21:00:37 DEBUG 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729696" 21:01:36 ERROR [http_cmd]: ERROR 21:01:36 ERROR --- status code --- 21:01:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:01:36 ERROR --- http response --- 21:01:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:01:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "instance": "10.142.0.107:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "instance": "10.142.0.43:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "100472297.33333334" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "11.621130520750715" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "ds-cts-2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "280342521.7933863" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-cts-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "ds-cts-2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "29.53333333333333" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "3321.7570990534596" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "ds-idrepo-2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "instance": "10.142.0.107:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "instance": "10.142.0.43:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0.3996333333333306" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "3596677120" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "197.054" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "ds-idrepo-2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "3316.657778962805" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "instance": "10.142.0.107:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "instance": "10.142.0.43:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "100472802.43333334" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "instance": "10.142.0.107:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "instance": "10.142.0.43:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "4605951144960" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "instance": "10.142.0.107:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "instance": "10.142.0.43:9100" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0.08066666666666149" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-cts-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "ds-cts-2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG [...] 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "ds-idrepo-2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "22.37662756633132" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "3333.988801493134" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-cts-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "ds-cts-2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0.5734333333333343" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "3316.1911745100656" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "0" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "ds-idrepo-0" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "973.5864753245983" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729696" 21:01:36 INFO [http_cmd]: http status code OK 21:01:36 DEBUG --- status code --- 21:01:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:01:36 DEBUG --- http response --- 21:01:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:01:36 DEBUG ----- output ----- 21:01:36 DEBUG { 21:01:36 DEBUG "status": "success", 21:01:36 DEBUG "data": { 21:01:36 DEBUG "resultType": "vector", 21:01:36 DEBUG "result": [ 21:01:36 DEBUG { 21:01:36 DEBUG "metric": { 21:01:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG [...] 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG 21:01:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:01:36 DEBUG }, 21:01:36 DEBUG "value": [ 21:01:36 DEBUG 1713729696, 21:01:36 DEBUG "599407.5" 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG ] 21:01:36 DEBUG } 21:01:36 DEBUG } 21:01:37 INFO 21:01:37 INFO [loop_until]: kubectl --namespace=xlou top pods 21:01:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:01:37 INFO [loop_until]: OK (rc = 0) 21:01:37 DEBUG --- stdout --- 21:01:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22290m 8805Mi am-767f6b67b5-qcqz2 21774m 8998Mi am-767f6b67b5-vbltz 24183m 8867Mi ds-cts-0 20014m 12145Mi ds-cts-1 20578m 12228Mi ds-cts-2 20327m 12075Mi ds-idrepo-0 9603m 4027Mi ds-idrepo-1 9567m 3996Mi ds-idrepo-2 9845m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11543m 3336Mi 21:01:37 DEBUG --- stderr --- 21:01:37 DEBUG 21:01:37 INFO 21:01:37 INFO [loop_until]: kubectl --namespace=xlou top node 21:01:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:01:37 INFO [loop_until]: OK (rc = 0) 21:01:37 DEBUG --- stdout --- 21:01:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6301Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10021m 33% 7872Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22316m 74% 11630Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3348Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20101m 67% 15417Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5286Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22349m 74% 11914Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19734m 66% 3219Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24688m 82% 11182Mi 10% gke-xlou-cdm-ds-0f601751-j488 10351m 34% 7450Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19362m 64% 17336Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10091m 33% 8039Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20434m 68% 14790Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11905m 74% 13569Mi 23% 21:01:37 DEBUG --- stderr --- 21:01:37 DEBUG 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "instance": "10.142.0.107:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "instance": "10.142.0.43:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "99843394.73333333" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729756" 21:02:36 ERROR [http_cmd]: ERROR 21:02:36 ERROR --- status code --- 21:02:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:02:36 ERROR --- http response --- 21:02:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:02:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "11.687884363427013" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-idrepo-0" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-cts-2" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "0" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-idrepo-1" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-idrepo-2" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "168088415.1718472" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "3828.533333333333" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:36 DEBUG "data": { 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG { 21:02:36 DEBUG "result": [ 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG { 21:02:36 DEBUG "data": { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "pod": "ds-idrepo-0" 21:02:36 DEBUG "result": [ 21:02:36 DEBUG }, 21:02:36 DEBUG { 21:02:36 DEBUG "value": [ 21:02:36 DEBUG "metric": { 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-cts-0" 21:02:36 DEBUG 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG }, 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG "value": [ 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG [...] 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:36 DEBUG [...] 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG "pod": "ds-idrepo-2" 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG 21:02:36 DEBUG { 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 DEBUG }, 21:02:36 DEBUG { 21:02:36 DEBUG 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "value": [ 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG 21:02:36 DEBUG "data": { 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "data": { 21:02:36 DEBUG "pod": "ds-cts-2" 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "0" 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG }, 21:02:36 DEBUG "result": [ 21:02:36 DEBUG ] 21:02:36 DEBUG "result": [ 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729756" 21:02:36 DEBUG "value": [ 21:02:36 DEBUG { 21:02:36 DEBUG } 21:02:36 DEBUG { 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "metric": { 21:02:36 DEBUG ] 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "30.4" 21:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:36 DEBUG } 21:02:36 DEBUG "instance": "10.142.0.107:9100" 21:02:36 DEBUG ] 21:02:36 DEBUG }, 21:02:36 DEBUG } 21:02:36 DEBUG }, 21:02:36 DEBUG } 21:02:36 DEBUG "value": [ 21:02:36 DEBUG "value": [ 21:02:36 DEBUG ] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG } 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG } 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:36 DEBUG "instance": "10.142.0.43:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "198.6125" 21:02:36 DEBUG "0.4136333333333823" 21:02:36 DEBUG ] 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "3690795008" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "instance": "10.142.0.107:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "instance": "10.142.0.43:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "99843628.26666667" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-idrepo-0" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-idrepo-2" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "0" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "3328.2666666666664" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-cts-0" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-cts-2" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "0" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "0" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "instance": "10.142.0.107:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "instance": "10.142.0.43:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "4611134299136" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "instance": "10.142.0.107:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "instance": "10.142.0.43:9100" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "0.08399999999998992" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "3323.266666666667" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG "result": [ 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 DEBUG { 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-idrepo-1" 21:02:36 DEBUG "pod": "ds-idrepo-0" 21:02:36 DEBUG }, 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-idrepo-2" 21:02:36 DEBUG "pod": "ds-idrepo-2" 21:02:36 DEBUG }, 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "9.657785944132861" 21:02:36 DEBUG "0" 21:02:36 DEBUG ] 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-cts-0" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-cts-2" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "0.5862999999999999" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "3327.2666666666664" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-idrepo-1" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-idrepo-2" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "4907.572230590169" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729756" 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "ds-idrepo-1" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "ds-idrepo-2" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "0" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:36 INFO [http_cmd]: http status code OK 21:02:36 DEBUG --- status code --- 21:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:02:36 DEBUG --- http response --- 21:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:02:36 DEBUG ----- output ----- 21:02:36 DEBUG { 21:02:36 DEBUG "status": "success", 21:02:36 DEBUG "data": { 21:02:36 DEBUG "resultType": "vector", 21:02:36 DEBUG "result": [ 21:02:36 DEBUG { 21:02:36 DEBUG "metric": { 21:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG [...] 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG 21:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:02:36 DEBUG }, 21:02:36 DEBUG "value": [ 21:02:36 DEBUG 1713729756, 21:02:36 DEBUG "599647" 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG ] 21:02:36 DEBUG } 21:02:36 DEBUG } 21:02:37 INFO 21:02:37 INFO [loop_until]: kubectl --namespace=xlou top pods 21:02:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:02:37 INFO [loop_until]: OK (rc = 0) 21:02:37 DEBUG --- stdout --- 21:02:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22435m 8800Mi am-767f6b67b5-qcqz2 22428m 9010Mi am-767f6b67b5-vbltz 24982m 8884Mi ds-cts-0 20298m 12145Mi ds-cts-1 20243m 12236Mi ds-cts-2 20172m 12082Mi ds-idrepo-0 9772m 4026Mi ds-idrepo-1 9488m 3996Mi ds-idrepo-2 9641m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11302m 3425Mi 21:02:37 DEBUG --- stderr --- 21:02:37 DEBUG 21:02:37 INFO 21:02:37 INFO [loop_until]: kubectl --namespace=xlou top node 21:02:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:02:37 INFO [loop_until]: OK (rc = 0) 21:02:37 DEBUG --- stdout --- 21:02:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6306Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9974m 33% 7862Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22337m 74% 11629Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3353Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20146m 67% 15448Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5281Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22784m 76% 11925Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19721m 66% 3225Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24906m 83% 11195Mi 10% gke-xlou-cdm-ds-0f601751-j488 10349m 34% 7440Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19767m 66% 17335Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10081m 33% 8046Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20290m 67% 14800Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11503m 72% 13661Mi 23% 21:02:37 DEBUG --- stderr --- 21:02:37 DEBUG 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "instance": "10.142.0.107:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "instance": "10.142.0.43:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "100811509.4" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729816" 21:03:36 ERROR [http_cmd]: ERROR 21:03:36 ERROR --- status code --- 21:03:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:03:36 ERROR --- http response --- 21:03:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:03:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "43111031.70256986" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "11.64809823203384" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "ds-idrepo-0" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "ds-cts-2" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "3682.633333333333" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "ds-idrepo-0" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "ds-idrepo-2" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:36 DEBUG "result": [ 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG "result": [ 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG { 21:03:36 DEBUG { 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "pod": "ds-idrepo-0" 21:03:36 DEBUG "data": { 21:03:36 DEBUG "pod": "ds-cts-0" 21:03:36 DEBUG "data": { 21:03:36 DEBUG }, 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG }, 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "value": [ 21:03:36 DEBUG "result": [ 21:03:36 DEBUG "value": [ 21:03:36 DEBUG "result": [ 21:03:36 DEBUG 21:03:36 DEBUG { 21:03:36 DEBUG 21:03:36 DEBUG { 21:03:36 DEBUG 21:03:36 DEBUG "metric": { 21:03:36 DEBUG 21:03:36 DEBUG "metric": { 21:03:36 DEBUG 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:36 DEBUG 21:03:36 DEBUG "instance": "10.142.0.107:9100" 21:03:36 DEBUG [...] 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG }, 21:03:36 DEBUG [...] 21:03:36 DEBUG }, 21:03:36 DEBUG 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "ds-idrepo-2" 21:03:36 DEBUG 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG "pod": "ds-cts-2" 21:03:36 DEBUG 21:03:36 DEBUG }, 21:03:36 DEBUG [...] 21:03:36 DEBUG { 21:03:36 DEBUG }, 21:03:36 DEBUG [...] 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG 21:03:36 DEBUG "data": { 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG 21:03:36 DEBUG "0" 21:03:36 DEBUG 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "32.03333333333333" 21:03:36 DEBUG 21:03:36 DEBUG ] 21:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:36 DEBUG "result": [ 21:03:36 DEBUG ] 21:03:36 DEBUG "instance": "10.142.0.43:9100" 21:03:36 DEBUG } 21:03:36 DEBUG }, 21:03:36 DEBUG { 21:03:36 DEBUG } 21:03:36 DEBUG }, 21:03:36 DEBUG ] 21:03:36 DEBUG "value": [ 21:03:36 DEBUG "metric": { 21:03:36 DEBUG ] 21:03:36 DEBUG "value": [ 21:03:36 DEBUG } 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:03:36 DEBUG } 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG } 21:03:36 DEBUG "203.138" 21:03:36 DEBUG }, 21:03:36 DEBUG } 21:03:36 DEBUG "0.4152666666666448" 21:03:36 DEBUG ] 21:03:36 DEBUG "value": [ 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 DEBUG [...] 21:03:36 DEBUG } 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "3709415424" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "instance": "10.142.0.107:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "instance": "10.142.0.43:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "100811992.69999999" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "3335.7999999999997" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "ds-cts-0" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "ds-cts-2" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "instance": "10.142.0.107:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "instance": "10.142.0.43:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "4616164869120" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0.0028775772770789944" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "instance": "10.142.0.107:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "instance": "10.142.0.43:9100" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0.08566666666666645" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "3334.3333333333335" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "ds-idrepo-0" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "ds-idrepo-2" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "ds-cts-0" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "ds-cts-2" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0.5146333333333322" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "0" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "3336.3333333333335" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "599650.5" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729816" 21:03:36 INFO [http_cmd]: http status code OK 21:03:36 DEBUG --- status code --- 21:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:03:36 DEBUG --- http response --- 21:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:03:36 DEBUG ----- output ----- 21:03:36 DEBUG { 21:03:36 DEBUG "status": "success", 21:03:36 DEBUG "data": { 21:03:36 DEBUG "resultType": "vector", 21:03:36 DEBUG "result": [ 21:03:36 DEBUG { 21:03:36 DEBUG "metric": { 21:03:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG [...] 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG 21:03:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:03:36 DEBUG }, 21:03:36 DEBUG "value": [ 21:03:36 DEBUG 1713729816, 21:03:36 DEBUG "1216.111899185168" 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG ] 21:03:36 DEBUG } 21:03:36 DEBUG } 21:03:37 INFO 21:03:37 INFO [loop_until]: kubectl --namespace=xlou top pods 21:03:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:03:37 INFO [loop_until]: OK (rc = 0) 21:03:37 DEBUG --- stdout --- 21:03:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22147m 8803Mi am-767f6b67b5-qcqz2 22903m 9024Mi am-767f6b67b5-vbltz 24090m 8876Mi ds-cts-0 19856m 12153Mi ds-cts-1 19754m 12231Mi ds-cts-2 20596m 12083Mi ds-idrepo-0 9549m 4027Mi ds-idrepo-1 9465m 3996Mi ds-idrepo-2 9572m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11706m 3546Mi 21:03:37 DEBUG --- stderr --- 21:03:37 DEBUG 21:03:37 INFO 21:03:37 INFO [loop_until]: kubectl --namespace=xlou top node 21:03:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:03:38 INFO [loop_until]: OK (rc = 0) 21:03:38 DEBUG --- stdout --- 21:03:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6306Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10007m 33% 7868Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22269m 74% 11624Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3357Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20309m 68% 15449Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5298Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22975m 76% 11931Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19805m 66% 3246Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25034m 83% 11178Mi 10% gke-xlou-cdm-ds-0f601751-j488 10326m 34% 7440Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19781m 66% 17352Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10136m 33% 8050Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20697m 69% 14796Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11921m 75% 13765Mi 23% 21:03:38 DEBUG --- stderr --- 21:03:38 DEBUG 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "instance": "10.142.0.107:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "instance": "10.142.0.43:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "101054112.96666667" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 ERROR [http_cmd]: ERROR 21:04:36 ERROR --- status code --- 21:04:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:04:36 ERROR --- http response --- 21:04:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:04:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "11.621195382577184" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG }, 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-cts-2" 21:04:36 DEBUG "pod": "ds-idrepo-2" 21:04:36 DEBUG }, 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0" 21:04:36 DEBUG "23109.402634937465" 21:04:36 DEBUG ] 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-idrepo-2" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-cts-0" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-cts-2" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "31.166666666666668" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "3647.0666666666666" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-idrepo-2" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:36 DEBUG "instance": "10.142.0.107:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG [...] 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:36 DEBUG "instance": "10.142.0.43:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG { 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "207.39" 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "0.38989999999993563" 21:04:36 DEBUG ] 21:04:36 DEBUG "data": { 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG "result": [ 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG { 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG "metric": { 21:04:36 DEBUG } 21:04:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "3815792640" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "instance": "10.142.0.107:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "instance": "10.142.0.43:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "101054319.93333334" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "3327.866666666667" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "3332.8333333333335" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:36 DEBUG "pod": "ds-cts-0" 21:04:36 DEBUG }, 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:36 DEBUG "pod": "ds-cts-2" 21:04:36 DEBUG }, 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0" 21:04:36 DEBUG "0" 21:04:36 DEBUG ] 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-idrepo-2" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "5.205492501644895" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG { 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-cts-0" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-cts-2" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0.5444999999999993" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "instance": "10.142.0.107:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "instance": "10.142.0.43:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "4621207659520" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "instance": "10.142.0.107:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "instance": "10.142.0.43:9100" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0.07500000000000757" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:36 DEBUG }, 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-idrepo-2" 21:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:36 DEBUG }, 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0" 21:04:36 DEBUG "3330.5666666666666" 21:04:36 DEBUG ] 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-idrepo-2" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "0" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "599772.5" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729876" 21:04:36 INFO [http_cmd]: http status code OK 21:04:36 DEBUG --- status code --- 21:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:04:36 DEBUG --- http response --- 21:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:04:36 DEBUG ----- output ----- 21:04:36 DEBUG { 21:04:36 DEBUG "status": "success", 21:04:36 DEBUG "data": { 21:04:36 DEBUG "resultType": "vector", 21:04:36 DEBUG "result": [ 21:04:36 DEBUG { 21:04:36 DEBUG "metric": { 21:04:36 DEBUG "pod": "ds-idrepo-0" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG [...] 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG 21:04:36 DEBUG "pod": "ds-idrepo-2" 21:04:36 DEBUG }, 21:04:36 DEBUG "value": [ 21:04:36 DEBUG 1713729876, 21:04:36 DEBUG "2077.249675050559" 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG ] 21:04:36 DEBUG } 21:04:36 DEBUG } 21:04:37 INFO 21:04:37 INFO [loop_until]: kubectl --namespace=xlou top pods 21:04:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:04:37 INFO [loop_until]: OK (rc = 0) 21:04:37 DEBUG --- stdout --- 21:04:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22578m 8805Mi am-767f6b67b5-qcqz2 22306m 9045Mi am-767f6b67b5-vbltz 24518m 8872Mi ds-cts-0 20784m 12179Mi ds-cts-1 20048m 12239Mi ds-cts-2 20010m 12090Mi ds-idrepo-0 9483m 4027Mi ds-idrepo-1 9502m 3996Mi ds-idrepo-2 9590m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11631m 3548Mi 21:04:37 DEBUG --- stderr --- 21:04:37 DEBUG 21:04:38 INFO 21:04:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:04:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:04:38 INFO [loop_until]: OK (rc = 0) 21:04:38 DEBUG --- stdout --- 21:04:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9961m 33% 7858Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22266m 74% 11643Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19560m 65% 15488Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5282Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22896m 76% 11963Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19744m 66% 3239Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24840m 83% 11181Mi 10% gke-xlou-cdm-ds-0f601751-j488 10350m 34% 7437Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19794m 66% 17352Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10090m 33% 8037Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20400m 68% 14805Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11875m 74% 13760Mi 23% 21:04:38 DEBUG --- stderr --- 21:04:38 DEBUG 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "instance": "10.142.0.107:9100" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "instance": "10.142.0.43:9100" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "101133332.63333333" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729936" 21:05:36 ERROR [http_cmd]: ERROR 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 ERROR --- status code --- 21:05:36 DEBUG --- status code --- 21:05:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 ERROR --- http response --- 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:05:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "1288.951048951049" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "ds-cts-2" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "11.715056079143206" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "ds-idrepo-2" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-cts-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "ds-cts-2" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "32.233333333333334" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "ds-idrepo-2" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "3656.866666666667" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "209.9345" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "3820883968" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "instance": "10.142.0.107:9100" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "instance": "10.142.0.43:9100" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0.398699999999917" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "instance": "10.142.0.107:9100" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "instance": "10.142.0.43:9100" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "101133363.76666667" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "3328.5" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "3331.9666666666667" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG { 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:36 DEBUG "instance": "10.142.0.107:9100" 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG }, 21:05:36 DEBUG { 21:05:36 DEBUG "value": [ 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG 21:05:36 DEBUG "data": { 21:05:36 DEBUG 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG 21:05:36 DEBUG "result": [ 21:05:36 DEBUG [...] 21:05:36 DEBUG { 21:05:36 DEBUG 21:05:36 DEBUG "metric": { 21:05:36 DEBUG 21:05:36 DEBUG "pod": "ds-cts-0" 21:05:36 DEBUG 21:05:36 DEBUG }, 21:05:36 DEBUG "instance": "10.142.0.43:9100" 21:05:36 DEBUG "value": [ 21:05:36 DEBUG }, 21:05:36 DEBUG 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG 21:05:36 DEBUG "4626300623872" 21:05:36 DEBUG [...] 21:05:36 DEBUG ] 21:05:36 DEBUG 21:05:36 DEBUG } 21:05:36 DEBUG 21:05:36 DEBUG ] 21:05:36 DEBUG 21:05:36 DEBUG } 21:05:36 DEBUG "pod": "ds-cts-2" 21:05:36 DEBUG } 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:36 DEBUG }, 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG [...] 21:05:36 DEBUG [...] 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:05:36 DEBUG 21:05:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:36 DEBUG }, 21:05:36 DEBUG { 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "data": { 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0.003071923076922567" 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "3328.9666666666667" 21:05:36 DEBUG ] 21:05:36 DEBUG "result": [ 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG { 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG "metric": { 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG "instance": "10.142.0.107:9100" 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 DEBUG }, 21:05:36 DEBUG } 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "instance": "10.142.0.43:9100" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0.08833333333334203" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-cts-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "ds-cts-2" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0.5605999999999994" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "ds-idrepo-2" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "0" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "599731" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729936" 21:05:36 INFO [http_cmd]: http status code OK 21:05:36 DEBUG --- status code --- 21:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:05:36 DEBUG --- http response --- 21:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:05:36 DEBUG ----- output ----- 21:05:36 DEBUG { 21:05:36 DEBUG "status": "success", 21:05:36 DEBUG "data": { 21:05:36 DEBUG "resultType": "vector", 21:05:36 DEBUG "result": [ 21:05:36 DEBUG { 21:05:36 DEBUG "metric": { 21:05:36 DEBUG "pod": "ds-idrepo-0" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG [...] 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG 21:05:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:05:36 DEBUG }, 21:05:36 DEBUG "value": [ 21:05:36 DEBUG 1713729936, 21:05:36 DEBUG "2005.0349650349651" 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG ] 21:05:36 DEBUG } 21:05:36 DEBUG } 21:05:37 INFO 21:05:37 INFO [loop_until]: kubectl --namespace=xlou top pods 21:05:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:05:37 INFO [loop_until]: OK (rc = 0) 21:05:37 DEBUG --- stdout --- 21:05:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22235m 8806Mi am-767f6b67b5-qcqz2 22490m 9047Mi am-767f6b67b5-vbltz 24439m 8880Mi ds-cts-0 20493m 12183Mi ds-cts-1 19242m 12240Mi ds-cts-2 19745m 12094Mi ds-idrepo-0 9851m 4026Mi ds-idrepo-1 9428m 3996Mi ds-idrepo-2 9547m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11480m 3550Mi 21:05:37 DEBUG --- stderr --- 21:05:37 DEBUG 21:05:38 INFO 21:05:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:05:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:05:38 INFO [loop_until]: OK (rc = 0) 21:05:38 DEBUG --- stdout --- 21:05:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10113m 33% 7874Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22318m 74% 11631Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3350Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20463m 68% 15487Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5308Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22948m 76% 11949Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19792m 66% 3234Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25025m 83% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 10351m 34% 7438Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19468m 65% 17357Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10126m 33% 8054Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20209m 67% 14783Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11914m 74% 13761Mi 23% 21:05:38 DEBUG --- stderr --- 21:05:38 DEBUG 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "instance": "10.142.0.107:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "instance": "10.142.0.43:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "101357431" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713729996" 21:06:36 ERROR [http_cmd]: ERROR 21:06:36 ERROR --- status code --- 21:06:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:06:36 ERROR --- http response --- 21:06:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:06:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "11.655877307340587" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-idrepo-1" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "416557590.8837148" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-idrepo-0" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:36 DEBUG "result": [ 21:06:36 DEBUG "result": [ 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG { 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:36 DEBUG "pod": "ds-idrepo-0" 21:06:36 DEBUG "data": { 21:06:36 DEBUG }, 21:06:36 DEBUG }, 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "value": [ 21:06:36 DEBUG "value": [ 21:06:36 DEBUG "result": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG { 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "metric": { 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-0" 21:06:36 DEBUG [...] 21:06:36 DEBUG [...] 21:06:36 DEBUG }, 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:36 DEBUG "pod": "ds-idrepo-2" 21:06:36 DEBUG 21:06:36 DEBUG }, 21:06:36 DEBUG }, 21:06:36 DEBUG [...] 21:06:36 DEBUG "value": [ 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG 21:06:36 DEBUG "213.8845" 21:06:36 DEBUG "0" 21:06:36 DEBUG ] 21:06:36 DEBUG ] 21:06:36 DEBUG 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG ] 21:06:36 DEBUG ] 21:06:36 DEBUG }, 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 DEBUG "value": [ 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "29.966666666666665" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-idrepo-0" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-idrepo-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "3829.2000000000003" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:36 DEBUG "instance": "10.142.0.107:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713729996" 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:36 DEBUG "instance": "10.142.0.43:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "3328.2999999999997" 21:06:36 DEBUG "101357356.66666666" 21:06:36 DEBUG ] 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "3826024448" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "instance": "10.142.0.107:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "instance": "10.142.0.43:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0.38716666666663097" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "instance": "10.142.0.107:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "instance": "10.142.0.43:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "4631597810688" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "3331" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-cts-0" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "3328.3333333333335" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-idrepo-1" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "20.29610203276681" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "instance": "10.142.0.107:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "instance": "10.142.0.43:9100" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0.08066666666668236" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-cts-0" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0.5806999999999998" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-idrepo-1" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "75486195.65859905" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-idrepo-0" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-idrepo-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "0" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "600063" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713729996" 21:06:36 INFO [http_cmd]: http status code OK 21:06:36 DEBUG --- status code --- 21:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:06:36 DEBUG --- http response --- 21:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:06:36 DEBUG ----- output ----- 21:06:36 DEBUG { 21:06:36 DEBUG "status": "success", 21:06:36 DEBUG "data": { 21:06:36 DEBUG "resultType": "vector", 21:06:36 DEBUG "result": [ 21:06:36 DEBUG { 21:06:36 DEBUG "metric": { 21:06:36 DEBUG "pod": "ds-idrepo-1" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG [...] 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG 21:06:36 DEBUG "pod": "ds-cts-2" 21:06:36 DEBUG }, 21:06:36 DEBUG "value": [ 21:06:36 DEBUG 1713729996, 21:06:36 DEBUG "86061733.80184558" 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG ] 21:06:36 DEBUG } 21:06:36 DEBUG } 21:06:38 INFO 21:06:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:06:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:06:38 INFO [loop_until]: OK (rc = 0) 21:06:38 DEBUG --- stdout --- 21:06:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22153m 8804Mi am-767f6b67b5-qcqz2 22707m 9013Mi am-767f6b67b5-vbltz 24641m 8887Mi ds-cts-0 20511m 12184Mi ds-cts-1 19536m 12251Mi ds-cts-2 20200m 12096Mi ds-idrepo-0 9441m 4027Mi ds-idrepo-1 9397m 3996Mi ds-idrepo-2 9417m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11830m 3553Mi 21:06:38 DEBUG --- stderr --- 21:06:38 DEBUG 21:06:38 INFO 21:06:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:06:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:06:38 INFO [loop_until]: OK (rc = 0) 21:06:38 DEBUG --- stdout --- 21:06:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9963m 33% 7872Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22238m 74% 11635Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3345Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20324m 68% 15486Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 5274Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22876m 76% 11929Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 18829m 63% 3227Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25070m 83% 11183Mi 10% gke-xlou-cdm-ds-0f601751-j488 10010m 33% 7441Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19748m 66% 17363Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10087m 33% 8041Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20269m 67% 14815Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11854m 74% 13765Mi 23% 21:06:38 DEBUG --- stderr --- 21:06:38 DEBUG 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "instance": "10.142.0.107:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "instance": "10.142.0.43:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "101291676.6" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730056" 21:07:36 ERROR [http_cmd]: ERROR 21:07:36 ERROR --- status code --- 21:07:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:07:36 ERROR --- http response --- 21:07:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:07:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "11.644480585750014" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-idrepo-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "25249.788887012328" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-cts-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-idrepo-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "3329.4333333333334" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "217.4855" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-cts-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-cts-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "30.633333333333333" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-idrepo-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "3496.233333333333" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "instance": "10.142.0.107:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "instance": "10.142.0.43:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "101292799.8" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "3823202304" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "instance": "10.142.0.107:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "instance": "10.142.0.43:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0.4037666666666439" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "3327.133333333333" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "instance": "10.142.0.107:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "instance": "10.142.0.43:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "4636722709504" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-cts-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-cts-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG { 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "3329.4666666666667" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:07:36 DEBUG } 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-idrepo-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "9.733587203738978" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "instance": "10.142.0.107:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "instance": "10.142.0.43:9100" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0.07233333333332627" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-cts-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-cts-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0.5350666666666655" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-idrepo-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-idrepo-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "0" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "600128.5" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730056" 21:07:36 INFO [http_cmd]: http status code OK 21:07:36 DEBUG --- status code --- 21:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:07:36 DEBUG --- http response --- 21:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:07:36 DEBUG ----- output ----- 21:07:36 DEBUG { 21:07:36 DEBUG "status": "success", 21:07:36 DEBUG "data": { 21:07:36 DEBUG "resultType": "vector", 21:07:36 DEBUG "result": [ 21:07:36 DEBUG { 21:07:36 DEBUG "metric": { 21:07:36 DEBUG "pod": "ds-idrepo-0" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG [...] 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG 21:07:36 DEBUG "pod": "ds-idrepo-2" 21:07:36 DEBUG }, 21:07:36 DEBUG "value": [ 21:07:36 DEBUG 1713730056, 21:07:36 DEBUG "4496.537747002196" 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG ] 21:07:36 DEBUG } 21:07:36 DEBUG } 21:07:38 INFO 21:07:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:07:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:07:38 INFO [loop_until]: OK (rc = 0) 21:07:38 DEBUG --- stdout --- 21:07:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22111m 8805Mi am-767f6b67b5-qcqz2 22465m 9012Mi am-767f6b67b5-vbltz 24768m 8892Mi ds-cts-0 20141m 12188Mi ds-cts-1 20993m 12251Mi ds-cts-2 20231m 12108Mi ds-idrepo-0 9515m 4026Mi ds-idrepo-1 9311m 3996Mi ds-idrepo-2 9568m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 11466m 3555Mi 21:07:38 DEBUG --- stderr --- 21:07:38 DEBUG 21:07:38 INFO 21:07:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:07:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:07:38 INFO [loop_until]: OK (rc = 0) 21:07:38 DEBUG --- stdout --- 21:07:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6293Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10000m 33% 7865Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22384m 74% 11624Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3345Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 20074m 67% 15496Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5280Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22482m 75% 11926Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19689m 65% 3227Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24881m 83% 11181Mi 10% gke-xlou-cdm-ds-0f601751-j488 10351m 34% 7427Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19956m 66% 17369Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10071m 33% 8040Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20409m 68% 14819Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11854m 74% 13761Mi 23% 21:07:38 DEBUG --- stderr --- 21:07:38 DEBUG 21:07:52 WARNING File /tmp/lodemon_data-20240421_203524/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [21/Apr/2024 21:07:52] "GET /monitoring/average?start_time=20240421_193837&stop_time=20240421_200551 HTTP/1.1" 200 - 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "instance": "10.142.0.107:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "instance": "10.142.0.43:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "24712860.7" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730116" 21:08:36 ERROR [http_cmd]: ERROR 21:08:36 ERROR --- status code --- 21:08:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:08:36 ERROR --- http response --- 21:08:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:08:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "1.837252487465846" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-idrepo-1" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "lodemon-5cd77b55c5-wc625" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "1861816.682703506" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-idrepo-0" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "ds-cts-2" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-idrepo-0" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "ds-idrepo-2" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "219.9685" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "155.43333333333334" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-idrepo-0" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "ds-idrepo-2" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-cts-0" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "ds-cts-2" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "28.866666666666667" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "4158.433333333333" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "instance": "10.142.0.107:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "instance": "10.142.0.43:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "24710509.5" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "2718253056" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "instance": "10.142.0.107:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "instance": "10.142.0.43:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0.47493333333326476" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "157.8" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-cts-0" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "ds-cts-2" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG ] 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG } 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG ] 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "instance": "10.142.0.107:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "instance": "10.142.0.43:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "4641153920000" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "155.73333333333332" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-idrepo-1" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "lodemon-5cd77b55c5-wc625" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0.0014866151188707467" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "instance": "10.142.0.107:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "instance": "10.142.0.43:9100" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0.08233333333333992" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-cts-0" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "ds-cts-2" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "27.8716" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-idrepo-1" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "lodemon-5cd77b55c5-wc625" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: http status code OK 21:08:36 DEBUG --- status code --- 21:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:36 DEBUG --- http response --- 21:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:36 DEBUG ----- output ----- 21:08:36 DEBUG { 21:08:36 DEBUG "status": "success", 21:08:36 DEBUG "data": { 21:08:36 DEBUG "resultType": "vector", 21:08:36 DEBUG "result": [ 21:08:36 DEBUG { 21:08:36 DEBUG "metric": { 21:08:36 DEBUG "pod": "ds-idrepo-0" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG [...] 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG 21:08:36 DEBUG "pod": "ds-idrepo-2" 21:08:36 DEBUG }, 21:08:36 DEBUG "value": [ 21:08:36 DEBUG 1713730116, 21:08:36 DEBUG "0" 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG ] 21:08:36 DEBUG } 21:08:36 DEBUG } 21:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730116" 21:08:37 INFO [http_cmd]: http status code OK 21:08:37 DEBUG --- status code --- 21:08:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:37 DEBUG --- http response --- 21:08:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:08:37 DEBUG ----- output ----- 21:08:37 DEBUG { 21:08:37 DEBUG "status": "success", 21:08:37 DEBUG "data": { 21:08:37 DEBUG "resultType": "vector", 21:08:37 DEBUG "result": [ 21:08:37 DEBUG { 21:08:37 DEBUG "metric": { 21:08:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:08:37 DEBUG }, 21:08:37 DEBUG "value": [ 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG [...] 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:08:37 DEBUG }, 21:08:37 DEBUG "value": [ 21:08:37 DEBUG 1713730116, 21:08:37 DEBUG "473330.5" 21:08:37 DEBUG ] 21:08:37 DEBUG } 21:08:37 DEBUG ] 21:08:37 DEBUG } 21:08:37 DEBUG } 21:08:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730117" 21:08:37 INFO [http_cmd]: http status code OK 21:08:37 DEBUG --- status code --- 21:08:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:08:37 DEBUG --- http response --- 21:08:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:08:37 DEBUG ----- output ----- 21:08:37 DEBUG { 21:08:37 DEBUG "status": "success", 21:08:37 DEBUG "data": { 21:08:37 DEBUG "resultType": "vector", 21:08:37 DEBUG "result": [ 21:08:37 DEBUG { 21:08:37 DEBUG "metric": { 21:08:37 DEBUG "pod": "ds-idrepo-1" 21:08:37 DEBUG }, 21:08:37 DEBUG "value": [ 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG [...] 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG 21:08:37 DEBUG "pod": "lodemon-5cd77b55c5-wc625" 21:08:37 DEBUG }, 21:08:37 DEBUG "value": [ 21:08:37 DEBUG 1713730117, 21:08:37 DEBUG "0" 21:08:37 DEBUG ] 21:08:37 DEBUG } 21:08:37 DEBUG ] 21:08:37 DEBUG } 21:08:37 DEBUG } 21:08:38 INFO 21:08:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:08:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:08:38 INFO [loop_until]: OK (rc = 0) 21:08:38 DEBUG --- stdout --- 21:08:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3806m 8803Mi am-767f6b67b5-qcqz2 5077m 9007Mi am-767f6b67b5-vbltz 4026m 8876Mi ds-cts-0 19409m 12182Mi ds-cts-1 18708m 12249Mi ds-cts-2 19630m 12100Mi ds-idrepo-0 1414m 4026Mi ds-idrepo-1 1511m 3995Mi ds-idrepo-2 1057m 3947Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 2757m 3065Mi 21:08:38 DEBUG --- stderr --- 21:08:38 DEBUG 21:08:38 INFO 21:08:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:08:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:08:38 INFO [loop_until]: OK (rc = 0) 21:08:38 DEBUG --- stdout --- 21:08:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1515m 5% 7861Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4358m 14% 11654Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3343Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 18878m 63% 15488Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5293Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 4934m 16% 11938Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 271m 0% 3234Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 4765m 15% 11214Mi 10% gke-xlou-cdm-ds-0f601751-j488 1772m 5% 7440Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18429m 61% 17356Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 1153m 3% 8046Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19261m 64% 14792Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 3853m 24% 13288Mi 22% 21:08:38 DEBUG --- stderr --- 21:08:38 DEBUG 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "instance": "10.142.0.107:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "instance": "10.142.0.43:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "106377463.8" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "8.95282628842029" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730176" 21:09:36 ERROR [http_cmd]: ERROR 21:09:36 ERROR --- status code --- 21:09:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:09:36 ERROR --- http response --- 21:09:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:09:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "ds-idrepo-0" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "115951324.39935078" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "ds-idrepo-0" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "ds-cts-2" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "0" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "225.386" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "ds-idrepo-0" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "ds-idrepo-2" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "0" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "3397.5333333333333" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "ds-cts-0" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "ds-cts-2" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "29.733333333333334" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "ds-idrepo-0" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "ds-idrepo-2" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "0" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "4002.2" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "instance": "10.142.0.107:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "instance": "10.142.0.43:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "106377877.63333333" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "4097122304" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "instance": "10.142.0.107:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "instance": "10.142.0.43:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "0.39729999999998805" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "0" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "3398.7" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "ds-cts-0" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "ds-cts-2" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "0" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "instance": "10.142.0.107:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "instance": "10.142.0.43:9100" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "4645783772160" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "3398.0333333333333" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: http status code OK 21:09:36 DEBUG --- status code --- 21:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:36 DEBUG --- http response --- 21:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:09:36 DEBUG ----- output ----- 21:09:36 DEBUG { 21:09:36 DEBUG "status": "success", 21:09:36 DEBUG "data": { 21:09:36 DEBUG "resultType": "vector", 21:09:36 DEBUG "result": [ 21:09:36 DEBUG { 21:09:36 DEBUG "metric": { 21:09:36 DEBUG "pod": "ds-idrepo-0" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG [...] 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG 21:09:36 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 21:09:36 DEBUG }, 21:09:36 DEBUG "value": [ 21:09:36 DEBUG 1713730176, 21:09:36 DEBUG "27.18965981617064" 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG ] 21:09:36 DEBUG } 21:09:36 DEBUG } 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730176" 21:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730176" 21:09:37 INFO [http_cmd]: http status code OK 21:09:37 DEBUG --- status code --- 21:09:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:37 DEBUG --- http response --- 21:09:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:09:37 DEBUG ----- output ----- 21:09:37 DEBUG { 21:09:37 DEBUG "status": "success", 21:09:37 DEBUG "data": { 21:09:37 DEBUG "resultType": "vector", 21:09:37 DEBUG "result": [ 21:09:37 DEBUG { 21:09:37 DEBUG "metric": { 21:09:37 DEBUG "instance": "10.142.0.107:9100" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG [...] 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG "instance": "10.142.0.43:9100" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 1713730176, 21:09:37 DEBUG "0.08900000000001378" 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG } 21:09:37 INFO [http_cmd]: http status code OK 21:09:37 DEBUG --- status code --- 21:09:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:37 DEBUG --- http response --- 21:09:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:09:37 DEBUG ----- output ----- 21:09:37 DEBUG { 21:09:37 DEBUG "status": "success", 21:09:37 DEBUG "data": { 21:09:37 DEBUG "resultType": "vector", 21:09:37 DEBUG "result": [ 21:09:37 DEBUG { 21:09:37 DEBUG "metric": { 21:09:37 DEBUG "pod": "ds-cts-0" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG [...] 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG "pod": "ds-cts-2" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 1713730176, 21:09:37 DEBUG "0.614033333333333" 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG } 21:09:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730177" 21:09:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730177" 21:09:37 INFO [http_cmd]: http status code OK 21:09:37 DEBUG --- status code --- 21:09:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:37 DEBUG --- http response --- 21:09:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:09:37 DEBUG ----- output ----- 21:09:37 DEBUG { 21:09:37 DEBUG "status": "success", 21:09:37 DEBUG "data": { 21:09:37 DEBUG "resultType": "vector", 21:09:37 DEBUG "result": [ 21:09:37 DEBUG { 21:09:37 DEBUG "metric": { 21:09:37 DEBUG "pod": "ds-idrepo-0" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG [...] 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 1713730177, 21:09:37 DEBUG "0" 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG } 21:09:37 INFO [http_cmd]: http status code OK 21:09:37 DEBUG --- status code --- 21:09:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:37 DEBUG --- http response --- 21:09:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:37 DEBUG ----- output ----- 21:09:37 DEBUG { 21:09:37 DEBUG "status": "success", 21:09:37 DEBUG "data": { 21:09:37 DEBUG "resultType": "vector", 21:09:37 DEBUG "result": [ 21:09:37 DEBUG { 21:09:37 DEBUG "metric": { 21:09:37 DEBUG "pod": "ds-idrepo-0" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG [...] 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG "pod": "ds-idrepo-2" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 1713730177, 21:09:37 DEBUG "0" 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG } 21:09:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730177" 21:09:37 INFO [http_cmd]: http status code OK 21:09:37 DEBUG --- status code --- 21:09:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:37 DEBUG --- http response --- 21:09:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:09:37 DEBUG ----- output ----- 21:09:37 DEBUG { 21:09:37 DEBUG "status": "success", 21:09:37 DEBUG "data": { 21:09:37 DEBUG "resultType": "vector", 21:09:37 DEBUG "result": [ 21:09:37 DEBUG { 21:09:37 DEBUG "metric": { 21:09:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG [...] 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 1713730177, 21:09:37 DEBUG "405382.5" 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG } 21:09:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730177" 21:09:37 INFO [http_cmd]: http status code OK 21:09:37 DEBUG --- status code --- 21:09:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:09:37 DEBUG --- http response --- 21:09:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:09:37 DEBUG ----- output ----- 21:09:37 DEBUG { 21:09:37 DEBUG "status": "success", 21:09:37 DEBUG "data": { 21:09:37 DEBUG "resultType": "vector", 21:09:37 DEBUG "result": [ 21:09:37 DEBUG { 21:09:37 DEBUG "metric": { 21:09:37 DEBUG "pod": "ds-idrepo-0" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG [...] 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG 21:09:37 DEBUG "pod": "overseer-0-68c5d5b884-p97w5" 21:09:37 DEBUG }, 21:09:37 DEBUG "value": [ 21:09:37 DEBUG 1713730177, 21:09:37 DEBUG "0" 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG ] 21:09:37 DEBUG } 21:09:37 DEBUG } 21:09:38 INFO 21:09:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:09:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:09:38 INFO [loop_until]: OK (rc = 0) 21:09:38 DEBUG --- stdout --- 21:09:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 24816m 8810Mi am-767f6b67b5-qcqz2 24426m 9049Mi am-767f6b67b5-vbltz 25358m 8933Mi ds-cts-0 20044m 12188Mi ds-cts-1 19520m 12261Mi ds-cts-2 20473m 12114Mi ds-idrepo-0 10626m 4028Mi ds-idrepo-1 10397m 3996Mi ds-idrepo-2 10250m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 13817m 4797Mi 21:09:38 DEBUG --- stderr --- 21:09:38 DEBUG 21:09:38 INFO 21:09:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:09:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:09:38 INFO [loop_until]: OK (rc = 0) 21:09:38 DEBUG --- stdout --- 21:09:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6304Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10660m 35% 7872Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 24698m 82% 11647Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3349Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20006m 67% 15502Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5308Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 24873m 83% 11980Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 23970m 80% 3465Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 25554m 85% 11292Mi 10% gke-xlou-cdm-ds-0f601751-j488 11030m 36% 7442Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19681m 65% 17387Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10656m 35% 8033Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20413m 68% 14823Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 13876m 87% 15329Mi 26% 21:09:38 DEBUG --- stderr --- 21:09:38 DEBUG 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730236" 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "instance": "10.142.0.107:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "instance": "10.142.0.43:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "93414952.26666667" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "13.507147935821301" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730236" 21:10:36 ERROR [http_cmd]: ERROR 21:10:36 ERROR --- status code --- 21:10:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:10:36 ERROR --- http response --- 21:10:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:10:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "ds-idrepo-0" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "ds-cts-2" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "0" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "ds-idrepo-0" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "0" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "230.8985" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "ds-idrepo-0" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "ds-idrepo-2" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "0" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "2597.9333333333334" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "ds-cts-0" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "ds-cts-2" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "27.133333333333333" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "ds-idrepo-0" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "ds-idrepo-2" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "0" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "3974.5" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "0" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730236" 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730236" 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "instance": "10.142.0.107:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "instance": "10.142.0.43:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "93414430" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "7633240064" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "instance": "10.142.0.107:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "instance": "10.142.0.43:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "0.37886666666661883" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "2565.0333333333333" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730236" 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "pod": "ds-cts-0" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "pod": "ds-cts-2" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "0" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: http status code OK 21:10:36 DEBUG --- status code --- 21:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:36 DEBUG --- http response --- 21:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:36 DEBUG ----- output ----- 21:10:36 DEBUG { 21:10:36 DEBUG "status": "success", 21:10:36 DEBUG "data": { 21:10:36 DEBUG "resultType": "vector", 21:10:36 DEBUG "result": [ 21:10:36 DEBUG { 21:10:36 DEBUG "metric": { 21:10:36 DEBUG "instance": "10.142.0.107:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG [...] 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG 21:10:36 DEBUG "instance": "10.142.0.43:9100" 21:10:36 DEBUG }, 21:10:36 DEBUG "value": [ 21:10:36 DEBUG 1713730236, 21:10:36 DEBUG "4650840593408" 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG ] 21:10:36 DEBUG } 21:10:36 DEBUG } 21:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730236" 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730236, 21:10:37 DEBUG "2597.6666666666665" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730237" 21:10:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730237" 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "instance": "10.142.0.107:9100" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "instance": "10.142.0.43:9100" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730237, 21:10:37 DEBUG "0.06766666666666954" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "pod": "ds-idrepo-0" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730237, 21:10:37 DEBUG "0" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730237" 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "pod": "ds-idrepo-0" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "pod": "ds-idrepo-2" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730237, 21:10:37 DEBUG "0" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730237" 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "pod": "ds-cts-0" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "pod": "ds-cts-2" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730237, 21:10:37 DEBUG "0.37090000000000034" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730237" 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "pod": "ds-idrepo-0" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730237, 21:10:37 DEBUG "0" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730237" 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730237, 21:10:37 DEBUG "398206" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730237" 21:10:37 INFO [http_cmd]: http status code OK 21:10:37 DEBUG --- status code --- 21:10:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:10:37 DEBUG --- http response --- 21:10:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:10:37 DEBUG ----- output ----- 21:10:37 DEBUG { 21:10:37 DEBUG "status": "success", 21:10:37 DEBUG "data": { 21:10:37 DEBUG "resultType": "vector", 21:10:37 DEBUG "result": [ 21:10:37 DEBUG { 21:10:37 DEBUG "metric": { 21:10:37 DEBUG "pod": "ds-idrepo-0" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG [...] 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG 21:10:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:10:37 DEBUG }, 21:10:37 DEBUG "value": [ 21:10:37 DEBUG 1713730237, 21:10:37 DEBUG "462.03600710178335" 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG ] 21:10:37 DEBUG } 21:10:37 DEBUG } 21:10:38 INFO 21:10:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:10:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:10:38 INFO [loop_until]: OK (rc = 0) 21:10:38 DEBUG --- stdout --- 21:10:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17524m 8814Mi am-767f6b67b5-qcqz2 18359m 9026Mi am-767f6b67b5-vbltz 19308m 8886Mi ds-cts-0 19595m 12191Mi ds-cts-1 19246m 12256Mi ds-cts-2 19791m 12119Mi ds-idrepo-0 7635m 4027Mi ds-idrepo-1 7736m 3996Mi ds-idrepo-2 7423m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 14624m 7774Mi 21:10:38 DEBUG --- stderr --- 21:10:38 DEBUG 21:10:38 INFO 21:10:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:10:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:10:38 INFO [loop_until]: OK (rc = 0) 21:10:38 DEBUG --- stdout --- 21:10:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7590m 25% 7876Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17890m 59% 11675Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3355Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19669m 65% 15502Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5289Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17954m 60% 11948Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 17008m 56% 4013Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 20847m 69% 11213Mi 10% gke-xlou-cdm-ds-0f601751-j488 7966m 26% 7445Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19307m 64% 17368Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 8079m 27% 8050Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20299m 67% 14825Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 15340m 96% 18022Mi 30% 21:10:38 DEBUG --- stderr --- 21:10:38 DEBUG 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730296" 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "instance": "10.142.0.107:9100" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "instance": "10.142.0.43:9100" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "51015487.33333333" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730296" 21:11:36 ERROR [http_cmd]: ERROR 21:11:36 ERROR --- status code --- 21:11:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:11:36 ERROR --- http response --- 21:11:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:11:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "14.550604858787894" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "pod": "ds-idrepo-0" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "pod": "ds-cts-2" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "0" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "pod": "ds-idrepo-0" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "270.4880142640164" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:36 DEBUG "pod": "ds-idrepo-0" 21:11:36 DEBUG }, 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:36 DEBUG "pod": "ds-idrepo-2" 21:11:36 DEBUG }, 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "232.63549999999998" 21:11:36 DEBUG "0" 21:11:36 DEBUG ] 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "1951.8666666666666" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "pod": "ds-cts-0" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "pod": "ds-cts-2" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "27.4" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "instance": "10.142.0.107:9100" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "instance": "10.142.0.43:9100" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "0.4202999999999652" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: http status code OK 21:11:36 DEBUG --- status code --- 21:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:36 DEBUG --- http response --- 21:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:36 DEBUG ----- output ----- 21:11:36 DEBUG { 21:11:36 DEBUG "status": "success", 21:11:36 DEBUG "data": { 21:11:36 DEBUG "resultType": "vector", 21:11:36 DEBUG "result": [ 21:11:36 DEBUG { 21:11:36 DEBUG "metric": { 21:11:36 DEBUG "pod": "ds-idrepo-0" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG [...] 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG 21:11:36 DEBUG "pod": "ds-idrepo-2" 21:11:36 DEBUG }, 21:11:36 DEBUG "value": [ 21:11:36 DEBUG 1713730296, 21:11:36 DEBUG "0" 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG ] 21:11:36 DEBUG } 21:11:36 DEBUG } 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730296" 21:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730296" 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:37 DEBUG }, 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:37 DEBUG }, 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730296, 21:11:37 DEBUG 1713730296, 21:11:37 DEBUG "0" 21:11:37 DEBUG "322.1" 21:11:37 DEBUG ] 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730297" 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730296, 21:11:37 DEBUG "1951.1" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730296, 21:11:37 DEBUG "11247030272" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "instance": "10.142.0.107:9100" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "instance": "10.142.0.43:9100" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "51015095.4" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730297" 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "instance": "10.142.0.107:9100" 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "ds-idrepo-0" 21:11:37 DEBUG }, 21:11:37 DEBUG "pod": "ds-cts-0" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "instance": "10.142.0.43:9100" 21:11:37 DEBUG 21:11:37 DEBUG "pod": "ds-idrepo-2" 21:11:37 DEBUG }, 21:11:37 DEBUG "pod": "ds-cts-2" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "4654982073344" 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "0" 21:11:37 DEBUG ] 21:11:37 DEBUG "0" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "1950.9666666666667" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730297" 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "ds-idrepo-0" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "0.0032620352638182016" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "instance": "10.142.0.107:9100" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "instance": "10.142.0.43:9100" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "0.056333333333327566" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "ds-idrepo-0" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "0" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "ds-cts-0" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "ds-cts-2" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "29.10243333333333" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730297" 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "ds-idrepo-0" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "idm-b85f9d978-wnhxb" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "2028.660106980123" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:37 INFO [http_cmd]: http status code OK 21:11:37 DEBUG --- status code --- 21:11:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:11:37 DEBUG --- http response --- 21:11:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:11:37 DEBUG ----- output ----- 21:11:37 DEBUG { 21:11:37 DEBUG "status": "success", 21:11:37 DEBUG "data": { 21:11:37 DEBUG "resultType": "vector", 21:11:37 DEBUG "result": [ 21:11:37 DEBUG { 21:11:37 DEBUG "metric": { 21:11:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG [...] 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG 21:11:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:11:37 DEBUG }, 21:11:37 DEBUG "value": [ 21:11:37 DEBUG 1713730297, 21:11:37 DEBUG "450022" 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG ] 21:11:37 DEBUG } 21:11:37 DEBUG } 21:11:38 INFO 21:11:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:11:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:11:38 INFO [loop_until]: OK (rc = 0) 21:11:38 DEBUG --- stdout --- 21:11:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 12993m 8803Mi am-767f6b67b5-qcqz2 11562m 9010Mi am-767f6b67b5-vbltz 11458m 8864Mi ds-cts-0 18737m 12191Mi ds-cts-1 18397m 12260Mi ds-cts-2 19201m 12115Mi ds-idrepo-0 5029m 4026Mi ds-idrepo-1 5786m 3995Mi ds-idrepo-2 5792m 3947Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 15145m 11325Mi 21:11:38 DEBUG --- stderr --- 21:11:38 DEBUG 21:11:38 INFO 21:11:38 INFO [loop_until]: kubectl --namespace=xlou top node 21:11:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:11:38 INFO [loop_until]: OK (rc = 0) 21:11:38 DEBUG --- stdout --- 21:11:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4900m 16% 7885Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11192m 37% 11644Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3364Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18685m 62% 15506Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5285Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11156m 37% 11932Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 12963m 43% 4083Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 12440m 41% 11197Mi 10% gke-xlou-cdm-ds-0f601751-j488 5130m 17% 7433Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18355m 61% 17384Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 5379m 18% 8052Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19192m 64% 14832Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 14762m 92% 22202Mi 37% 21:11:38 DEBUG --- stderr --- 21:11:38 DEBUG 21:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730356" 21:12:36 INFO [http_cmd]: http status code OK 21:12:36 DEBUG --- status code --- 21:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:36 DEBUG --- http response --- 21:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:36 DEBUG ----- output ----- 21:12:36 DEBUG { 21:12:36 DEBUG "status": "success", 21:12:36 DEBUG "data": { 21:12:36 DEBUG "resultType": "vector", 21:12:36 DEBUG "result": [ 21:12:36 DEBUG { 21:12:36 DEBUG "metric": { 21:12:36 DEBUG "instance": "10.142.0.107:9100" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG [...] 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG "instance": "10.142.0.43:9100" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 1713730356, 21:12:36 DEBUG "67397693.33333333" 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG } 21:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730356" 21:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730356" 21:12:36 INFO [http_cmd]: http status code OK 21:12:36 DEBUG --- status code --- 21:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:36 DEBUG --- http response --- 21:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:36 DEBUG ----- output ----- 21:12:36 DEBUG { 21:12:36 DEBUG "status": "success", 21:12:36 DEBUG "data": { 21:12:36 DEBUG "resultType": "vector", 21:12:36 DEBUG "result": [ 21:12:36 DEBUG { 21:12:36 DEBUG "metric": { 21:12:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG [...] 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 1713730356, 21:12:36 DEBUG "14.88884658597473" 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG } 21:12:36 ERROR [http_cmd]: ERROR 21:12:36 ERROR --- status code --- 21:12:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:12:36 ERROR --- http response --- 21:12:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:12:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730356" 21:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730356" 21:12:36 INFO [http_cmd]: http status code OK 21:12:36 DEBUG --- status code --- 21:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:36 DEBUG --- http response --- 21:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:12:36 DEBUG ----- output ----- 21:12:36 DEBUG { 21:12:36 DEBUG "status": "success", 21:12:36 DEBUG "data": { 21:12:36 DEBUG "resultType": "vector", 21:12:36 DEBUG "result": [ 21:12:36 DEBUG { 21:12:36 DEBUG "metric": { 21:12:36 DEBUG "pod": "ds-idrepo-0" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG [...] 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG "pod": "ds-idrepo-2" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 1713730356, 21:12:36 DEBUG "0" 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG } 21:12:36 INFO [http_cmd]: http status code OK 21:12:36 DEBUG --- status code --- 21:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:36 DEBUG --- http response --- 21:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:36 DEBUG ----- output ----- 21:12:36 DEBUG { 21:12:36 DEBUG "status": "success", 21:12:36 DEBUG "data": { 21:12:36 DEBUG "resultType": "vector", 21:12:36 DEBUG "result": [ 21:12:36 DEBUG { 21:12:36 DEBUG "metric": { 21:12:36 DEBUG "pod": "ds-idrepo-0" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG [...] 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG 21:12:36 DEBUG "pod": "ds-cts-2" 21:12:36 DEBUG }, 21:12:36 DEBUG "value": [ 21:12:36 DEBUG 1713730356, 21:12:36 DEBUG "0" 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG ] 21:12:36 DEBUG } 21:12:36 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "ds-idrepo-0" 21:12:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:37 DEBUG }, 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "ds-idrepo-2" 21:12:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:37 DEBUG }, 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0" 21:12:37 DEBUG "233.49099999999999" 21:12:37 DEBUG ] 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "1646.4333333333334" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG { 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:37 DEBUG "data": { 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "pod": "ds-cts-0" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG }, 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "value": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "instance": "10.142.0.107:9100" 21:12:37 DEBUG 21:12:37 DEBUG "pod": "ds-idrepo-0" 21:12:37 DEBUG }, 21:12:37 DEBUG 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG [...] 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG "pod": "ds-cts-2" 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG }, 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG 21:12:37 DEBUG "instance": "10.142.0.43:9100" 21:12:37 DEBUG "30.2" 21:12:37 DEBUG "pod": "ds-idrepo-2" 21:12:37 DEBUG }, 21:12:37 DEBUG ] 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG } 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG ] 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0.4583666666666659" 21:12:37 DEBUG } 21:12:37 DEBUG "0" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "1645.5333333333333" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "3739.4666666666667" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "14839599104" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "instance": "10.142.0.107:9100" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "instance": "10.142.0.43:9100" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "67396631.36666666" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "instance": "10.142.0.107:9100" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "instance": "10.142.0.43:9100" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0.12300000000000559" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG { 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "result": [ 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG { 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "ds-idrepo-0" 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "ds-cts-0" 21:12:37 DEBUG }, 21:12:37 DEBUG "pod": "ds-idrepo-0" 21:12:37 DEBUG "instance": "10.142.0.107:9100" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG }, 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG "value": [ 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "ds-idrepo-2" 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "ds-cts-2" 21:12:37 DEBUG }, 21:12:37 DEBUG "pod": "ds-idrepo-2" 21:12:37 DEBUG "instance": "10.142.0.43:9100" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG }, 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0" 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0" 21:12:37 DEBUG ] 21:12:37 DEBUG "0" 21:12:37 DEBUG "4659582706688" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "1646.6666666666667" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "ds-cts-0" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "ds-cts-2" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0.22496666666666557" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "ds-idrepo-0" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "lodemon-5cd77b55c5-wc625" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "0" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730357" 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "ds-idrepo-0" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "lodemon-5cd77b55c5-wc625" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "4579.637472471624" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:37 INFO [http_cmd]: http status code OK 21:12:37 DEBUG --- status code --- 21:12:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:12:37 DEBUG --- http response --- 21:12:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:12:37 DEBUG ----- output ----- 21:12:37 DEBUG { 21:12:37 DEBUG "status": "success", 21:12:37 DEBUG "data": { 21:12:37 DEBUG "resultType": "vector", 21:12:37 DEBUG "result": [ 21:12:37 DEBUG { 21:12:37 DEBUG "metric": { 21:12:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG [...] 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG 21:12:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:12:37 DEBUG }, 21:12:37 DEBUG "value": [ 21:12:37 DEBUG 1713730357, 21:12:37 DEBUG "442207" 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG ] 21:12:37 DEBUG } 21:12:37 DEBUG } 21:12:38 INFO 21:12:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:12:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:12:38 INFO [loop_until]: OK (rc = 0) 21:12:38 DEBUG --- stdout --- 21:12:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 12247m 8807Mi am-767f6b67b5-qcqz2 13555m 9008Mi am-767f6b67b5-vbltz 12102m 8867Mi ds-cts-0 18801m 12191Mi ds-cts-1 19543m 12258Mi ds-cts-2 19738m 12114Mi ds-idrepo-0 5123m 4027Mi ds-idrepo-1 4903m 3995Mi ds-idrepo-2 5216m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 14760m 15933Mi 21:12:38 DEBUG --- stderr --- 21:12:38 DEBUG 21:12:39 INFO 21:12:39 INFO [loop_until]: kubectl --namespace=xlou top node 21:12:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:12:39 INFO [loop_until]: OK (rc = 0) 21:12:39 DEBUG --- stdout --- 21:12:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5006m 16% 7879Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12512m 41% 11639Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3351Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19333m 64% 15495Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5304Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12241m 41% 11943Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 9955m 33% 4241Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 14197m 47% 11184Mi 10% gke-xlou-cdm-ds-0f601751-j488 5132m 17% 7440Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18673m 62% 17362Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 5439m 18% 8034Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19623m 65% 14795Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 15112m 95% 25221Mi 43% 21:12:39 DEBUG --- stderr --- 21:12:39 DEBUG 21:13:36 WARNING File /tmp/lodemon_data-20240421_203524/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [21/Apr/2024 21:13:36] "GET /monitoring/average?start_time=20240421_200952&stop_time=20240421_201135 HTTP/1.1" 200 - 21:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730416" 21:13:36 INFO [http_cmd]: http status code OK 21:13:36 DEBUG --- status code --- 21:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:36 DEBUG --- http response --- 21:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:36 DEBUG ----- output ----- 21:13:36 DEBUG { 21:13:36 DEBUG "status": "success", 21:13:36 DEBUG "data": { 21:13:36 DEBUG "resultType": "vector", 21:13:36 DEBUG "result": [ 21:13:36 DEBUG { 21:13:36 DEBUG "metric": { 21:13:36 DEBUG "instance": "10.142.0.107:9100" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG [...] 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG "instance": "10.142.0.43:9100" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 1713730416, 21:13:36 DEBUG "76092153.36666667" 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG } 21:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730416" 21:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730416" 21:13:36 ERROR [http_cmd]: ERROR 21:13:36 ERROR --- status code --- 21:13:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:13:36 ERROR --- http response --- 21:13:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:13:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:13:36 INFO [http_cmd]: http status code OK 21:13:36 DEBUG --- status code --- 21:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:36 DEBUG --- http response --- 21:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:36 DEBUG ----- output ----- 21:13:36 DEBUG { 21:13:36 DEBUG "status": "success", 21:13:36 DEBUG "data": { 21:13:36 DEBUG "resultType": "vector", 21:13:36 DEBUG "result": [ 21:13:36 DEBUG { 21:13:36 DEBUG "metric": { 21:13:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG [...] 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 1713730416, 21:13:36 DEBUG "14.908119016939136" 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG } 21:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730416" 21:13:36 INFO [http_cmd]: http status code OK 21:13:36 DEBUG --- status code --- 21:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:36 DEBUG --- http response --- 21:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:36 DEBUG ----- output ----- 21:13:36 DEBUG { 21:13:36 DEBUG "status": "success", 21:13:36 DEBUG "data": { 21:13:36 DEBUG "resultType": "vector", 21:13:36 DEBUG "result": [ 21:13:36 DEBUG { 21:13:36 DEBUG "metric": { 21:13:36 DEBUG "pod": "ds-idrepo-0" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG [...] 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG "pod": "ds-cts-2" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 1713730416, 21:13:36 DEBUG "0" 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG } 21:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730416" 21:13:36 INFO [http_cmd]: http status code OK 21:13:36 DEBUG --- status code --- 21:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:36 DEBUG --- http response --- 21:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:13:36 DEBUG ----- output ----- 21:13:36 DEBUG { 21:13:36 DEBUG "status": "success", 21:13:36 DEBUG "data": { 21:13:36 DEBUG "resultType": "vector", 21:13:36 DEBUG "result": [ 21:13:36 DEBUG { 21:13:36 DEBUG "metric": { 21:13:36 DEBUG "pod": "ds-idrepo-0" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG [...] 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG 21:13:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:13:36 DEBUG }, 21:13:36 DEBUG "value": [ 21:13:36 DEBUG 1713730416, 21:13:36 DEBUG "32355.788079000973" 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG ] 21:13:36 DEBUG } 21:13:36 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG { 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "data": { 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG "result": [ 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG { 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "ds-idrepo-0" 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:37 DEBUG }, 21:13:37 DEBUG }, 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG "value": [ 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG { 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "data": { 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG [...] 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "result": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG { 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "metric": { 21:13:37 DEBUG 21:13:37 DEBUG "pod": "ds-idrepo-2" 21:13:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:37 DEBUG }, 21:13:37 DEBUG }, 21:13:37 DEBUG }, 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG "value": [ 21:13:37 DEBUG "value": [ 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "0" 21:13:37 DEBUG "234.9835" 21:13:37 DEBUG 21:13:37 DEBUG "1528.4" 21:13:37 DEBUG ] 21:13:37 DEBUG ] 21:13:37 DEBUG 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG [...] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG ] 21:13:37 DEBUG 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG 21:13:37 DEBUG } 21:13:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "1525.4" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG { 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG "data": { 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG "result": [ 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG { 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "pod": "ds-cts-0" 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG }, 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG "value": [ 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "ds-idrepo-0" 21:13:37 DEBUG 21:13:37 DEBUG "instance": "10.142.0.107:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG [...] 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "ds-cts-2" 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG }, 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "34.233333333333334" 21:13:37 DEBUG 21:13:37 DEBUG "pod": "ds-idrepo-2" 21:13:37 DEBUG ] 21:13:37 DEBUG "instance": "10.142.0.43:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG } 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG ] 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG } 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "0" 21:13:37 DEBUG } 21:13:37 DEBUG "0.4077666666667331" 21:13:37 DEBUG ] 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG { 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG "result": [ 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:37 DEBUG "data": { 21:13:37 DEBUG }, 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "value": [ 21:13:37 DEBUG "result": [ 21:13:37 DEBUG 21:13:37 DEBUG { 21:13:37 DEBUG 21:13:37 DEBUG "metric": { 21:13:37 DEBUG 21:13:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:37 DEBUG [...] 21:13:37 DEBUG }, 21:13:37 DEBUG 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:37 DEBUG 21:13:37 DEBUG }, 21:13:37 DEBUG [...] 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG "3134.6666666666665" 21:13:37 DEBUG 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG ] 21:13:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG } 21:13:37 DEBUG }, 21:13:37 DEBUG ] 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:37 DEBUG "value": [ 21:13:37 DEBUG } 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG } 21:13:37 DEBUG { 21:13:37 DEBUG "0" 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG ] 21:13:37 DEBUG "data": { 21:13:37 DEBUG } 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG ] 21:13:37 DEBUG "result": [ 21:13:37 DEBUG } 21:13:37 DEBUG { 21:13:37 DEBUG } 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "16977170432" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "instance": "10.142.0.107:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "instance": "10.142.0.43:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "76091883.56666666" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "instance": "10.142.0.107:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "instance": "10.142.0.43:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 DEBUG "4663969205248" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "ds-cts-0" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "ds-cts-2" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "29.7489" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "ds-idrepo-0" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "6.478137264842204" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "instance": "10.142.0.107:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "instance": "10.142.0.43:9100" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "0.07466666666667833" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "ds-cts-0" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "ds-cts-2" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "0" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:37 DEBUG }, 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG "value": [ 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG "1527.8" 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG ] 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG } 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG ] 21:13:37 DEBUG { 21:13:37 DEBUG } 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG } 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "ds-idrepo-0" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "ds-idrepo-2" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "0" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "ds-idrepo-0" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "0" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730417" 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "363505.5" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:37 INFO [http_cmd]: http status code OK 21:13:37 DEBUG --- status code --- 21:13:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:13:37 DEBUG --- http response --- 21:13:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:13:37 DEBUG ----- output ----- 21:13:37 DEBUG { 21:13:37 DEBUG "status": "success", 21:13:37 DEBUG "data": { 21:13:37 DEBUG "resultType": "vector", 21:13:37 DEBUG "result": [ 21:13:37 DEBUG { 21:13:37 DEBUG "metric": { 21:13:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG [...] 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG 21:13:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:13:37 DEBUG }, 21:13:37 DEBUG "value": [ 21:13:37 DEBUG 1713730417, 21:13:37 DEBUG "0" 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG ] 21:13:37 DEBUG } 21:13:37 DEBUG } 21:13:38 INFO 21:13:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:13:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:13:38 INFO [loop_until]: OK (rc = 0) 21:13:38 DEBUG --- stdout --- 21:13:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 13049m 8803Mi am-767f6b67b5-qcqz2 13667m 9012Mi am-767f6b67b5-vbltz 12297m 8867Mi ds-cts-0 19366m 12191Mi ds-cts-1 19058m 12254Mi ds-cts-2 19520m 12117Mi ds-idrepo-0 4457m 4027Mi ds-idrepo-1 5671m 3995Mi ds-idrepo-2 4671m 3947Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1181Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-68c5d5b884-p97w5 12680m 16Mi 21:13:38 DEBUG --- stderr --- 21:13:38 DEBUG 21:13:39 INFO 21:13:39 INFO [loop_until]: kubectl --namespace=xlou top node 21:13:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:13:39 INFO [loop_until]: OK (rc = 0) 21:13:39 DEBUG --- stdout --- 21:13:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6301Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4262m 14% 7873Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10797m 36% 11678Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3356Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19020m 63% 15491Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 5279Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10009m 33% 11974Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 11637m 38% 3822Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 13285m 44% 11216Mi 10% gke-xlou-cdm-ds-0f601751-j488 3986m 13% 7430Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18915m 63% 17361Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 5268m 17% 8043Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19619m 65% 14820Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11829m 74% 10412Mi 17% 21:13:39 DEBUG --- stderr --- 21:13:39 DEBUG 21:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730476" 21:14:36 INFO [http_cmd]: http status code OK 21:14:36 DEBUG --- status code --- 21:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:36 DEBUG --- http response --- 21:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:36 DEBUG ----- output ----- 21:14:36 DEBUG { 21:14:36 DEBUG "status": "success", 21:14:36 DEBUG "data": { 21:14:36 DEBUG "resultType": "vector", 21:14:36 DEBUG "result": [ 21:14:36 DEBUG { 21:14:36 DEBUG "metric": { 21:14:36 DEBUG "instance": "10.142.0.107:9100" 21:14:36 DEBUG }, 21:14:36 DEBUG "value": [ 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG [...] 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG "instance": "10.142.0.43:9100" 21:14:36 DEBUG }, 21:14:36 DEBUG "value": [ 21:14:36 DEBUG 1713730476, 21:14:36 DEBUG "16524596.666666668" 21:14:36 DEBUG ] 21:14:36 DEBUG } 21:14:36 DEBUG ] 21:14:36 DEBUG } 21:14:36 DEBUG } 21:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730476" 21:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730476" 21:14:36 ERROR [http_cmd]: ERROR 21:14:36 ERROR --- status code --- 21:14:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:14:36 ERROR --- http response --- 21:14:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:14:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:14:36 INFO [http_cmd]: http status code OK 21:14:36 DEBUG --- status code --- 21:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:36 DEBUG --- http response --- 21:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:36 DEBUG ----- output ----- 21:14:36 DEBUG { 21:14:36 DEBUG "status": "success", 21:14:36 DEBUG "data": { 21:14:36 DEBUG "resultType": "vector", 21:14:36 DEBUG "result": [ 21:14:36 DEBUG { 21:14:36 DEBUG "metric": { 21:14:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:14:36 DEBUG }, 21:14:36 DEBUG "value": [ 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG [...] 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:14:36 DEBUG }, 21:14:36 DEBUG "value": [ 21:14:36 DEBUG 1713730476, 21:14:36 DEBUG "0.03373398049100939" 21:14:36 DEBUG ] 21:14:36 DEBUG } 21:14:36 DEBUG ] 21:14:36 DEBUG } 21:14:36 DEBUG } 21:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730476" 21:14:36 INFO [http_cmd]: http status code OK 21:14:36 DEBUG --- status code --- 21:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:36 DEBUG --- http response --- 21:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:36 DEBUG ----- output ----- 21:14:36 DEBUG { 21:14:36 DEBUG "status": "success", 21:14:36 DEBUG "data": { 21:14:36 DEBUG "resultType": "vector", 21:14:36 DEBUG "result": [ 21:14:36 DEBUG { 21:14:36 DEBUG "metric": { 21:14:36 DEBUG "pod": "ds-idrepo-0" 21:14:36 DEBUG }, 21:14:36 DEBUG "value": [ 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG 21:14:36 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-cts-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730476, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-idrepo-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-idrepo-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-idrepo-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "236.633" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0.1" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-idrepo-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-idrepo-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-cts-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-cts-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "30.366666666666667" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "instance": "10.142.0.107:9100" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "instance": "10.142.0.43:9100" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0.4537999999999859" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "instance": "10.142.0.107:9100" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "instance": "10.142.0.43:9100" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "16482900.7" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-cts-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-cts-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "28.501666666666665" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG } 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG { 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG "data": { 21:14:37 DEBUG "data": { 21:14:37 DEBUG { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG "result": [ 21:14:37 DEBUG "data": { 21:14:37 DEBUG { 21:14:37 DEBUG { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "result": [ 21:14:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:37 DEBUG { 21:14:37 DEBUG }, 21:14:37 DEBUG }, 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "value": [ 21:14:37 DEBUG "value": [ 21:14:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG }, 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG [...] 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:37 DEBUG 21:14:37 DEBUG }, 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG }, 21:14:37 DEBUG 21:14:37 DEBUG "value": [ 21:14:37 DEBUG { 21:14:37 DEBUG "value": [ 21:14:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG }, 21:14:37 DEBUG "2287.3" 21:14:37 DEBUG "data": { 21:14:37 DEBUG "0" 21:14:37 DEBUG "value": [ 21:14:37 DEBUG ] 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG ] 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG } 21:14:37 DEBUG "result": [ 21:14:37 DEBUG } 21:14:37 DEBUG "112709632" 21:14:37 DEBUG ] 21:14:37 DEBUG { 21:14:37 DEBUG ] 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG "metric": { 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 DEBUG "instance": "10.142.0.107:9100" 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG }, 21:14:37 DEBUG } 21:14:37 DEBUG "value": [ 21:14:37 DEBUG } 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "instance": "10.142.0.43:9100" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "4667484746752" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "instance": "10.142.0.107:9100" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "instance": "10.142.0.43:9100" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0.0546666666666719" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-idrepo-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-idrepo-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-idrepo-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "ds-cts-0" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-cts-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "0" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-cts-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "89133623.29085667" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "261379" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730477" 21:14:37 INFO [http_cmd]: http status code OK 21:14:37 DEBUG --- status code --- 21:14:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:14:37 DEBUG --- http response --- 21:14:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:14:37 DEBUG ----- output ----- 21:14:37 DEBUG { 21:14:37 DEBUG "status": "success", 21:14:37 DEBUG "data": { 21:14:37 DEBUG "resultType": "vector", 21:14:37 DEBUG "result": [ 21:14:37 DEBUG { 21:14:37 DEBUG "metric": { 21:14:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG [...] 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG 21:14:37 DEBUG "pod": "ds-cts-2" 21:14:37 DEBUG }, 21:14:37 DEBUG "value": [ 21:14:37 DEBUG 1713730477, 21:14:37 DEBUG "55831351.37816638" 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG ] 21:14:37 DEBUG } 21:14:37 DEBUG } 21:14:38 INFO 21:14:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:14:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:14:38 INFO [loop_until]: OK (rc = 0) 21:14:38 DEBUG --- stdout --- 21:14:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2498m 8730Mi am-767f6b67b5-qcqz2 2499m 8940Mi am-767f6b67b5-vbltz 2792m 8791Mi ds-cts-0 17829m 12186Mi ds-cts-1 17250m 12266Mi ds-cts-2 19064m 12115Mi ds-idrepo-0 790m 4029Mi ds-idrepo-1 786m 3998Mi ds-idrepo-2 697m 3948Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1182Mi idm-b85f9d978-wnhxb 5m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 582m 350Mi 21:14:38 DEBUG --- stderr --- 21:14:38 DEBUG 21:14:39 INFO 21:14:39 INFO [loop_until]: kubectl --namespace=xlou top node 21:14:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:14:39 INFO [loop_until]: OK (rc = 0) 21:14:39 DEBUG --- stdout --- 21:14:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 6306Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 862m 2% 7853Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2464m 8% 11578Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 144m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17902m 59% 15499Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5296Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2526m 8% 11875Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 150m 0% 3719Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 2855m 9% 11131Mi 9% gke-xlou-cdm-ds-0f601751-j488 818m 2% 7421Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17684m 59% 17335Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 913m 3% 8042Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19297m 64% 14810Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 561m 3% 10613Mi 18% 21:14:39 DEBUG --- stderr --- 21:14:39 DEBUG 21:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730536" 21:15:36 INFO [http_cmd]: http status code OK 21:15:36 DEBUG --- status code --- 21:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:36 DEBUG --- http response --- 21:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:36 DEBUG ----- output ----- 21:15:36 DEBUG { 21:15:36 DEBUG "status": "success", 21:15:36 DEBUG "data": { 21:15:36 DEBUG "resultType": "vector", 21:15:36 DEBUG "result": [ 21:15:36 DEBUG { 21:15:36 DEBUG "metric": { 21:15:36 DEBUG "instance": "10.142.0.107:9100" 21:15:36 DEBUG }, 21:15:36 DEBUG "value": [ 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG [...] 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG "instance": "10.142.0.43:9100" 21:15:36 DEBUG }, 21:15:36 DEBUG "value": [ 21:15:36 DEBUG 1713730536, 21:15:36 DEBUG "15477754.933333334" 21:15:36 DEBUG ] 21:15:36 DEBUG } 21:15:36 DEBUG ] 21:15:36 DEBUG } 21:15:36 DEBUG } 21:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730536" 21:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730536" 21:15:36 ERROR [http_cmd]: ERROR 21:15:36 ERROR --- status code --- 21:15:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:15:36 ERROR --- http response --- 21:15:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:15:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:15:36 INFO [http_cmd]: http status code OK 21:15:36 DEBUG --- status code --- 21:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:36 DEBUG --- http response --- 21:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:36 DEBUG ----- output ----- 21:15:36 DEBUG { 21:15:36 DEBUG "status": "success", 21:15:36 DEBUG "data": { 21:15:36 DEBUG "resultType": "vector", 21:15:36 DEBUG "result": [ 21:15:36 DEBUG { 21:15:36 DEBUG "metric": { 21:15:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:15:36 DEBUG }, 21:15:36 DEBUG "value": [ 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG [...] 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG 21:15:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:15:36 DEBUG }, 21:15:36 DEBUG "value": [ 21:15:36 DEBUG 1713730536, 21:15:36 DEBUG "1.1121393096806824" 21:15:36 DEBUG ] 21:15:36 DEBUG } 21:15:36 DEBUG ] 21:15:36 DEBUG } 21:15:36 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-idrepo-0" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-cts-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-idrepo-1" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-idrepo-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "312266.69379726663" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-idrepo-0" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-idrepo-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "63.06666666666666" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "238.4115" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "instance": "10.142.0.107:9100" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "instance": "10.142.0.43:9100" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0.5101999999999691" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-idrepo-0" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-idrepo-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "8.9" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-cts-0" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-cts-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "29.733333333333334" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "602550272" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730537" 21:15:37 DEBUG "result": [ 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG { 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:37 DEBUG }, 21:15:37 DEBUG "data": { 21:15:37 DEBUG "value": [ 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG 21:15:37 DEBUG "result": [ 21:15:37 DEBUG 21:15:37 DEBUG { 21:15:37 DEBUG 21:15:37 DEBUG "metric": { 21:15:37 DEBUG [...] 21:15:37 DEBUG "instance": "10.142.0.107:9100" 21:15:37 DEBUG 21:15:37 DEBUG }, 21:15:37 DEBUG 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:37 DEBUG 21:15:37 DEBUG }, 21:15:37 DEBUG 21:15:37 DEBUG "value": [ 21:15:37 DEBUG [...] 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG 21:15:37 DEBUG "0" 21:15:37 DEBUG 21:15:37 DEBUG ] 21:15:37 DEBUG 21:15:37 DEBUG } 21:15:37 DEBUG "instance": "10.142.0.43:9100" 21:15:37 DEBUG ] 21:15:37 DEBUG }, 21:15:37 DEBUG } 21:15:37 DEBUG "value": [ 21:15:37 DEBUG } 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "4671231439872" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "instance": "10.142.0.107:9100" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "instance": "10.142.0.43:9100" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "15475308.466666665" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-cts-0" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-cts-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "25.338366666666666" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "2313.366666666667" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "instance": "10.142.0.107:9100" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "instance": "10.142.0.43:9100" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0.10933333333332297" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-idrepo-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0.8535721504387702" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-idrepo-0" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-idrepo-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "6.7" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "ds-cts-0" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-cts-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-idrepo-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "0" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "137430" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730537" 21:15:37 INFO [http_cmd]: http status code OK 21:15:37 DEBUG --- status code --- 21:15:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:15:37 DEBUG --- http response --- 21:15:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 21:15:37 DEBUG ----- output ----- 21:15:37 DEBUG { 21:15:37 DEBUG "status": "success", 21:15:37 DEBUG "data": { 21:15:37 DEBUG "resultType": "vector", 21:15:37 DEBUG "result": [ 21:15:37 DEBUG { 21:15:37 DEBUG "metric": { 21:15:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG [...] 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG 21:15:37 DEBUG "pod": "ds-idrepo-2" 21:15:37 DEBUG }, 21:15:37 DEBUG "value": [ 21:15:37 DEBUG 1713730537, 21:15:37 DEBUG "8473.42896869807" 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG ] 21:15:37 DEBUG } 21:15:37 DEBUG } 21:15:38 INFO 21:15:38 INFO [loop_until]: kubectl --namespace=xlou top pods 21:15:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:15:39 INFO [loop_until]: OK (rc = 0) 21:15:39 DEBUG --- stdout --- 21:15:39 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2302m 8732Mi am-767f6b67b5-qcqz2 2327m 8931Mi am-767f6b67b5-vbltz 2414m 8788Mi ds-cts-0 18707m 12183Mi ds-cts-1 17404m 12081Mi ds-cts-2 18607m 12108Mi ds-idrepo-0 1096m 4029Mi ds-idrepo-1 788m 4008Mi ds-idrepo-2 673m 3969Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1011m 751Mi 21:15:39 DEBUG --- stderr --- 21:15:39 DEBUG 21:15:39 INFO 21:15:39 INFO [loop_until]: kubectl --namespace=xlou top node 21:15:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:15:39 INFO [loop_until]: OK (rc = 0) 21:15:39 DEBUG --- stdout --- 21:15:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6297Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1103m 3% 7851Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2281m 7% 11591Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3359Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18544m 62% 15459Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5325Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2356m 7% 11860Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 78m 0% 3651Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 2546m 8% 11119Mi 9% gke-xlou-cdm-ds-0f601751-j488 812m 2% 7439Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17295m 57% 17159Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 850m 2% 8026Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18496m 61% 14804Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 11280Mi 19% 21:15:39 DEBUG --- stderr --- 21:15:39 DEBUG 21:16:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730596" 21:16:36 INFO [http_cmd]: http status code OK 21:16:36 DEBUG --- status code --- 21:16:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:36 DEBUG --- http response --- 21:16:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:36 DEBUG ----- output ----- 21:16:36 DEBUG { 21:16:36 DEBUG "status": "success", 21:16:36 DEBUG "data": { 21:16:36 DEBUG "resultType": "vector", 21:16:36 DEBUG "result": [ 21:16:36 DEBUG { 21:16:36 DEBUG "metric": { 21:16:36 DEBUG "instance": "10.142.0.107:9100" 21:16:36 DEBUG }, 21:16:36 DEBUG "value": [ 21:16:36 DEBUG 21:16:36 DEBUG 21:16:36 DEBUG 21:16:36 DEBUG [...] 21:16:36 DEBUG 21:16:36 DEBUG 21:16:36 DEBUG 21:16:36 DEBUG "instance": "10.142.0.43:9100" 21:16:36 DEBUG }, 21:16:36 DEBUG "value": [ 21:16:36 DEBUG 1713730596, 21:16:36 DEBUG "16077231.466666667" 21:16:36 DEBUG ] 21:16:36 DEBUG } 21:16:36 DEBUG ] 21:16:36 DEBUG } 21:16:36 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "1.0309568640559894" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730597" 21:16:37 ERROR [http_cmd]: ERROR 21:16:37 ERROR --- status code --- 21:16:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:16:37 ERROR --- http response --- 21:16:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:16:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-idrepo-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "ds-cts-2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-idrepo-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "36287.54505025038" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-idrepo-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "ds-idrepo-2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "63.06666666666666" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "instance": "10.142.0.107:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "instance": "10.142.0.43:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0.12070000000003346" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-idrepo-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "ds-idrepo-2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "8.9" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG --- status code --- 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730597" 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG { 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:16:37 DEBUG "data": { 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "data": { 21:16:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG }, 21:16:37 DEBUG "result": [ 21:16:37 DEBUG "value": [ 21:16:37 DEBUG { 21:16:37 DEBUG 21:16:37 DEBUG "metric": { 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:37 DEBUG 21:16:37 DEBUG }, 21:16:37 DEBUG [...] 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:37 DEBUG [...] 21:16:37 DEBUG }, 21:16:37 DEBUG 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG 21:16:37 DEBUG "0" 21:16:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:37 DEBUG ] 21:16:37 DEBUG }, 21:16:37 DEBUG } 21:16:37 DEBUG "value": [ 21:16:37 DEBUG ] 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG } 21:16:37 DEBUG "239.334" 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-cts-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "ds-cts-2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "23.406433333333332" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "1895452672" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-cts-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "ds-cts-2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "29.8" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730597" 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "instance": "10.142.0.107:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "instance": "10.142.0.43:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "4673671302144" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "instance": "10.142.0.107:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "instance": "10.142.0.43:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "16074719.2" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "1529.6333333333332" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "instance": "10.142.0.107:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "instance": "10.142.0.43:9100" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0.06633333333332415" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "1.3869942963813535" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-idrepo-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "ds-idrepo-2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "6.7" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "ds-cts-0" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "ds-cts-2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "0" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "23072.5" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730597" 21:16:37 INFO [http_cmd]: http status code OK 21:16:37 DEBUG --- status code --- 21:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:16:37 DEBUG --- http response --- 21:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 21:16:37 DEBUG ----- output ----- 21:16:37 DEBUG { 21:16:37 DEBUG "status": "success", 21:16:37 DEBUG "data": { 21:16:37 DEBUG "resultType": "vector", 21:16:37 DEBUG "result": [ 21:16:37 DEBUG { 21:16:37 DEBUG "metric": { 21:16:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG [...] 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG 21:16:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:16:37 DEBUG }, 21:16:37 DEBUG "value": [ 21:16:37 DEBUG 1713730597, 21:16:37 DEBUG "426.912294708828" 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG ] 21:16:37 DEBUG } 21:16:37 DEBUG } 21:16:39 INFO 21:16:39 INFO [loop_until]: kubectl --namespace=xlou top pods 21:16:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:16:39 INFO [loop_until]: OK (rc = 0) 21:16:39 DEBUG --- stdout --- 21:16:39 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1677m 8723Mi am-767f6b67b5-qcqz2 2316m 8925Mi am-767f6b67b5-vbltz 1905m 8787Mi ds-cts-0 18167m 12190Mi ds-cts-1 3650m 12078Mi ds-cts-2 18935m 12106Mi ds-idrepo-0 366m 4030Mi ds-idrepo-1 456m 4008Mi ds-idrepo-2 300m 3971Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 4m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 1003m 2211Mi 21:16:39 DEBUG --- stderr --- 21:16:39 DEBUG 21:16:39 INFO 21:16:39 INFO [loop_until]: kubectl --namespace=xlou top node 21:16:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:16:39 INFO [loop_until]: OK (rc = 0) 21:16:39 DEBUG --- stdout --- 21:16:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 324m 1% 7872Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1756m 5% 11579Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3374Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18327m 61% 15470Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 5309Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1672m 5% 11866Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 3614Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 1923m 6% 11123Mi 9% gke-xlou-cdm-ds-0f601751-j488 313m 1% 7454Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 4234m 14% 17219Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 483m 1% 8063Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18972m 63% 14817Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 12395Mi 21% 21:16:39 DEBUG --- stderr --- 21:16:39 DEBUG 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "instance": "10.142.0.107:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "instance": "10.142.0.43:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "121524.7" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0.667652352431774" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713730657" 21:17:37 ERROR [http_cmd]: ERROR 21:17:37 ERROR --- status code --- 21:17:37 ERROR http status code is 400 (expected [200, 401, 403, 404]) 21:17:37 ERROR --- http response --- 21:17:37 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 21:17:37 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-idrepo-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "ds-cts-2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-idrepo-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "348870886.1363976" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-idrepo-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "ds-idrepo-2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "instance": "10.142.0.107:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "instance": "10.142.0.43:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0.0007666666666409582" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "2407043072" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-idrepo-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "ds-idrepo-2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "239.838" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-cts-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "ds-cts-2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "17.308766666666667" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "instance": "10.142.0.107:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "instance": "10.142.0.43:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "4673875098624" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-cts-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "ds-cts-2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "29.766666666666666" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "instance": "10.142.0.107:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "instance": "10.142.0.43:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "118792.23333333334" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0.03333333333333333" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "instance": "10.142.0.107:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "instance": "10.142.0.43:9100" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0.0009999999999990906" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0.6155655126101197" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-idrepo-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "ds-idrepo-2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "ds-cts-0" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "ds-cts-2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "am-767f6b67b5-9zw22" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-vbltz" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "0.5" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713730657" 21:17:37 INFO [http_cmd]: http status code OK 21:17:37 DEBUG --- status code --- 21:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 21:17:37 DEBUG --- http response --- 21:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 21:17:37 DEBUG ----- output ----- 21:17:37 DEBUG { 21:17:37 DEBUG "status": "success", 21:17:37 DEBUG "data": { 21:17:37 DEBUG "resultType": "vector", 21:17:37 DEBUG "result": [ 21:17:37 DEBUG { 21:17:37 DEBUG "metric": { 21:17:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG [...] 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG 21:17:37 DEBUG "pod": "am-767f6b67b5-qcqz2" 21:17:37 DEBUG }, 21:17:37 DEBUG "value": [ 21:17:37 DEBUG 1713730657, 21:17:37 DEBUG "611.2520519325475" 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG ] 21:17:37 DEBUG } 21:17:37 DEBUG } 21:17:39 INFO 21:17:39 INFO [loop_until]: kubectl --namespace=xlou top pods 21:17:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:17:39 INFO [loop_until]: OK (rc = 0) 21:17:39 DEBUG --- stdout --- 21:17:39 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 641m 8713Mi am-767f6b67b5-qcqz2 606m 8917Mi am-767f6b67b5-vbltz 645m 8779Mi ds-cts-0 17256m 11381Mi ds-cts-1 16m 12077Mi ds-cts-2 18424m 12104Mi ds-idrepo-0 16m 4030Mi ds-idrepo-1 13m 4010Mi ds-idrepo-2 13m 3972Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1182Mi idm-b85f9d978-wnhxb 3m 1241Mi lodemon-5cd77b55c5-wc625 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-68c5d5b884-p97w5 525m 2630Mi 21:17:39 DEBUG --- stderr --- 21:17:39 DEBUG 21:17:39 INFO 21:17:39 INFO [loop_until]: kubectl --namespace=xlou top node 21:17:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 21:17:39 INFO [loop_until]: OK (rc = 0) 21:17:39 DEBUG --- stdout --- 21:17:39 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 75m 0% 7872Mi 7% gke-xlou-cdm-default-pool-1b0769c8-nhlf 724m 2% 11554Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3369Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17013m 56% 14640Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5284Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 727m 2% 11845Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 3577Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 640m 2% 11086Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7457Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 17210Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 8053Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 18358m 61% 14807Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 711m 4% 12999Mi 22% 21:17:39 DEBUG --- stderr --- 21:17:39 DEBUG 21:17:49 INFO Finished: True 21:17:49 INFO Waiting for threads to register finish flag 21:18:39 INFO Done. Have a nice day! :) 127.0.0.1 - - [21/Apr/2024 21:18:39] "GET /monitoring/stop HTTP/1.1" 200 - 21:18:42 WARNING File /tmp/lodemon_data-20240421_203524/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [21/Apr/2024 21:18:46] "GET /monitoring/process HTTP/1.1" 200 -